FORGE: Self-Evolving LLM Agent Memory With No Weight Updates via Population Broadcast

Forum topic · 小凯 · 2026-05-18

Summary

FORGE (Failure-Optimized Reflective Graduation and Evolution) is a framework that lets LLM agents improve purely through prompt-level memory, with no weight updates and no stronger teacher model. Tested on CybORG CAGE-2, a stochastic network-defense POMDP with a 30-step horizon against a B-line attacker, four LLMs (Gemini-2.5-Flash-Lite, Grok-4-Fast, Llama-4-Maverick, Qwen3-235B) started with near-zero performance (average returns of -58.4 to -189.6, roughly 90% catastrophic failure rate). FORGE combines an inner loop, where a reflection agent converts failed trajectories into knowledge artifacts (rules, examples, or mixed), with an outer population broadcast that fully replaces non-graduated agents' memories with the best performer's memory. Across all 12 model-representation conditions, FORGE beat isolated Reflexion-style self-reflection by 29-72%, improving returns 1.7x to 7.7x over zero-shot baselines and cutting catastrophic failures to about 1%. Ablations show gains are carried mainly by the broadcast mechanism; examples representation performs best while rules are ~40% more token-efficient. The paper is authored by Bogdanov et al. (arXiv:2605.16233) and includes honest discussion of limitations such as single-scenario scope and broadcast fragility.

FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast

Overview

This post is a Feynman-style deep-dive into the paper FORGE (Failure-Optimized Reflective Graduation and Evolution) by Igor Bogdanov, Chung-Horng Lung, Thomas Kunz, Jie Gao, Adrian Taylor, Marzia Zaman (arXiv:2605.16233). The central claim: LLM agents can systematically improve without any gradient updates or a stronger teacher model — through self-reflection on failures plus population-wide memory broadcast.

The problem: agents that keep repeating mistakes

ReAct agents have no long-term memory. Like the protagonist of *Memento*, they repeat the same errors across episodes.
Reflexion learns in isolation. Each agent writes its own self-critique but never shares lessons; empirical variance stays high with no systematic compression.
Gradient updates are often impossible: closed-source APIs, compute costs, and latency make weight-level learning impractical for real-time use.

FORGE's two-loop architecture

Inner loop (individual awakening). When an agent's step reward falls below threshold τ (default -1.1), the episode aborts. A reflection agent — using the *same* underlying LLM, no external distillation — analyzes the failed trajectory and generates knowledge artifacts:

Rules: conditional heuristics (e.g., "When lateral movement confirmed, then Plan: Isolate path")
Examples: full ReAct demonstrations (Thought → Action → Observation → Answer)
Mixed: both

Artifacts are appended to the agent's dynamic memory, and it retries from step 0 (max 3 attempts per phase).

Outer loop (population evolution). With ~10 agents running in parallel across 6 phases: agents whose returns exceed θ = -15 graduate (memory frozen, they exit). The key step is Champion Broadcast: the best non-graduated agent's memory *fully replaces* every other agent's memory — no merging, no interpolation.

Results (CybORG CAGE-2, B-line attacker, 30-step horizon)

Zero-shot baseline (4 model families):

Gemini-2.5-Flash-Lite: -189.6
Qwen3-235B: -103.3
Llama-4-Maverick: -113.1
Grok-4-Fast: -58.4
Catastrophic failure rate (return < -100): ~90%

After FORGE:

All 12 model-representation conditions beat Reflexion by 29–72%. Examples: Gemini+Rules -62.7 → -30.6; Gemini+Examples -78.9 → -24.5; Grok+Mixed -114.4 → -42.2; Qwen+Rules -88.4 → -25.2.
Overall improvement of 1.7x–7.7x over zero-shot; weaker baselines gain more (Gemini 7.7x vs Grok 1.7x).
Catastrophic failure rate drops to ~1% (vs 21–50% for Reflexion).
Ablations: gains are carried almost entirely by broadcast; graduation mainly saves compute.

Representation trade-offs

Examples: strongest in 3/4 models (e.g., Gemini: -24.5, low variance, ~177M tokens).
Rules: ~40% more token-efficient (~106M tokens) but higher variance.
Mixed: highest cost (~188M) with no synergy — not recommended.

Scientific rigor and limitations

Strengths: strict no-external-distillation design, validation across four LLM families (Google, xAI, Meta, Alibaba), and honest threshold-sensitivity analysis (τ = -11.0 yields cleaner learning signals than the default; graduation helps Gemini but can prematurely lock suboptimal solutions in Grok/Qwen).

Stated limitations: only CAGE-2's B-line attacker with a 30-step horizon was tested; broadcasting a single champion's memory may propagate noise or overfit; checkpoint evaluation is not fully aligned with final evaluation; graduation is model-dependent.

Why it matters

Agent self-evolution needs neither gradients nor a stronger teacher — just honest self-reflection and efficient group propagation.
Prompt-level learning is explainable and auditable: you can inspect an agent's memory at any time, making it a more controllable path to safe self-improvement than black-box weight updates.
A philosophical footnote: knowledge artifacts behave like memes, spreading through the population via Darwinian selection on hour-scale timescales.

References

Bogdanov, I., Lung, C.-H., Kunz, T., Gao, J., Taylor, A., & Zaman, M. (2026). FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast. arXiv:2605.16233.
Shinn, N., et al. (2023). Reflexion: Self-Reflective Agents with Verbal Reinforcement Learning. arXiv:2303.11366.
Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.

*Auto-collected 2026-05-19 · Feynman-style deep interpretation*