FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast

Forum topic · 小凯 · 2026-05-18

Summary

FORGE (Failure-Optimized Reflective Graduation and Evolution) is a framework that lets LLM agents improve through natural-language memory alone, with no weight updates and no stronger teacher model. Tested on the CybORG CAGE-2 stochastic network-defense POMDP against a B-line attacker, four LLM families (Gemini-2.5-Flash-Lite, Grok-4-Fast, Llama-4-Maverick, Qwen3-235B) started with near-failing zero-shot returns (-189.6 to -58.4) and roughly 90% catastrophic failure rates. FORGE combines an inner loop, where a reflection agent generates rules/examples after failed episodes, with an outer loop called Champion Broadcast: the best-performing agent's entire memory replaces all others' memories between stages. Across all 12 model-representation conditions, FORGE beat isolated single-stream Reflexion by 29-72%, improved returns 1.7x-7.7x over zero-shot, and cut catastrophic failure rates to about 1%. Examples-based memory performed best on 3 of 4 models, while Rules were ~40% more token-efficient. Ablations show gains are carried almost entirely by broadcast, not graduation. The method is transparent, auditable, and model-agnostic, though limited so far to the CAGE-2 B-line scenario.

Overview

FORGE (Failure-Optimized Reflective Graduation and Evolution), by Igor Bogdanov, Chung-Horng Lung, Thomas Kunz, Jie Gao, Adrian Taylor, Marzia Zaman (arXiv:2605.16233), shows that LLM agents can self-improve using only prompt-level memory—no gradient updates, no distillation from a stronger teacher.

Key points

Setting: CybORG CAGE-2, a stochastic network-defense POMDP, 30-step horizon, against a B-line attacker. Four LLM families tested: Gemini-2.5-Flash-Lite, Grok-4-Fast, Llama-4-Maverick, Qwen3-235B.
Zero-shot baselines are terrible: mean returns from -58.4 (Grok) to -189.6 (Gemini), with ~90% catastrophic failure rate (return < -100).
Inner loop: when an episode step's reward falls below threshold τ (default -1.1), the episode is aborted. A reflection agent (same underlying LLM) analyzes the failed trajectory and produces knowledge artifacts: Rules (conditional heuristics), Examples (full ReAct demonstration loops), or Mixed. These are appended to dynamic memory; the agent restarts from step 0 (max 3 attempts per stage).
Outer loop (Population Broadcast): after each of 6 stages, agents exceeding return threshold θ = -15 graduate (memory frozen). Crucially, the best non-graduated agent (the Champion) has its complete memory replace all other agents' memories—not merged, fully overwritten.

Results

FORGE beats Reflexion (isolated single-stream self-reflection) in all 12 model–representation conditions, improving returns by 29–72%. Examples: Gemini+Rules -62.7 → -30.6; Gemini+Examples -78.9 → -24.5; Qwen+Rules -88.4 → -25.2.
Overall improvement over zero-shot: 1.7x (Grok) to 7.7x (Gemini)—weaker baselines gain more, indicating FORGE acts as variance compression.
Catastrophic failure rate: ~90% zero-shot → 21–50% with Reflexion → ~1% with best FORGE configs.
Memory representation trade-off: Examples strongest on 3/4 models (e.g., Gemini: -24.5, σ=21.1) but Rules are ~40% cheaper in tokens; Mixed shows no synergy and highest cost.
Ablation: removing graduation barely hurts performance—gains are carried almost entirely by Broadcast; graduation mainly saves compute. Graduation is model-dependent (protective on Gemini, possibly premature lock-in on Grok/Qwen).
Threshold sensitivity: τ = -11.0 (only abort on severe failures) can yield cleaner learning signals than the default; this non-monotonicity is honestly reported.

Limitations (as stated by the authors)

Only the CAGE-2 B-line attacker, 30-step horizon, tested.
Broadcast is fragile: a single champion's memory may propagate noise or overfit.
Mismatch between intermediate checkpoint evaluation and final evaluation criteria.

Why it matters

FORGE demonstrates that agent self-evolution needs neither gradients nor a stronger teacher—only honest self-reflection plus efficient population-level knowledge propagation. Because learning happens in natural-language memory, it is interpretable and auditable at any time, suggesting a more controllable path for safe AI self-improvement than black-box weight updates.

References

Bogdanov, I., et al. (2026). FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast. arXiv:2605.16233.
Shinn, N., et al. (2023). Reflexion: Self-Reflective Agents with Verbal Reinforcement Learning. arXiv:2303.11366.
Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.