FORGE: Self-Evolving LLM Agent Memory Without Weight Updates (arXiv:2605.16233)

Forum topic · 小凯 · 2026-05-18

Summary

This article analyzes FORGE, a protocol by Igor Bogdanov et al. (arXiv:2605.16233) that enables large language model agents to self-improve in a cybersecurity environment purely through natural-language memory, with no weight updates. Tested on CybORG CAGE-2, a 30-step, partially observable cyber-defense game with scalar rewards, base LLMs perform poorly: Gemini-2.5-Flash-Lite scores -189.6 on average versus a random baseline of -154. FORGE runs 10 independent agent instances across 6 stages. Each instance follows a hierarchical ReAct architecture (Planner, Analyst, ActionChooser) with dynamic memory storing rules or examples; failures trigger reflector agents that add knowledge artifacts. After each stage, the champion's full memory is broadcast to all instances, replacing theirs, while instances scoring above a threshold (-15) graduate and are frozen. Results: Gemini improves 7.7x to -24.5, with Qwen (-24.3), Llama (-28.3), and Grok (-33.7) also improving; FORGE beats single-thread Reflexion by 29-72% across all combinations. Notably, weaker zero-shot models gain larger relative improvements, suggesting FORGE acts as a variance-reduction and capability-gap-bridging mechanism. Examples-based memory yields peak performance, while rules are ~40% cheaper in tokens.

Key points

This post is a deep dive into FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast (Bogdanov et al., arXiv:2605.16233), a prompt-only self-improvement protocol for LLM agents tested on the CybORG CAGE-2 cyber-defense environment.

The problem: a brutal environment for LLM agents

CAGE-2 casts the agent as a blue-team defender protecting a 13-host network against a red-team attacker following a standard kill chain (recon → intrusion → lateral movement → privilege escalation → control).
It features a long horizon (30 steps), partial observability, scalar rewards only, and stochastic attacker behavior.
Zero-shot LLMs fail badly: Gemini-2.5-Flash-Lite averages -189.6 (worse than random play at -154), Grok-4-Fast -58.4, Llama-4-Maverick -113.1, Qwen3-235B -103.3. Top RL algorithms reach around -3.47.

The core idea: population-level evolution of natural-language memory

Instead of gradient updates, FORGE adapts agent behavior purely by evolving prompt content, extending Reflexion-style verbal reinforcement learning:

1. Inner loop (failure anatomy): A hierarchical ReAct agent (Planner + Analyst + ActionChooser sub-agents) runs episodes. Any step whose reward falls below a threshold (τ = -1.1) aborts the episode; a reflector/exemplifier agent then writes a new rule or worked example into dynamic memory, and the episode restarts (up to 3 retries per checkpoint). 2. Outer loop (champion broadcast): N = 10 independent instances explore across S = 6 stages. After each stage, the best-scoring champion has its entire memory broadcast destructively to all other instances, fully replacing their memory — merging conflicting rule sets in text space is unreliable, so FORGE clones the best instead (a text-space analogue of Population-Based Training). 3. Graduation: Any instance scoring above θ = -15 in a checkpoint is frozen and protected from future overwrites, preserving rare good solutions while cutting compute as active instances drop out.

Results

| Model | Zero-shot | Best with FORGE | Improvement | |---|---|---|---| | Gemini-2.5-Flash-Lite | -189.6 | -24.5 | 7.7x | | Grok-4-Fast | -58.4 | -33.7 | 1.7x | | Llama-4-Maverick | -113.1 | -28.3 | 4.0x | | Qwen3-235B | -103.3 | -24.3 | 4.3x |

FORGE beats single-thread Reflexion by 29–72% on all 12 model × memory-representation combinations, showing the bottleneck is not reflection quality but a lack of selection pressure to propagate rare good discoveries.
Examples (few-shot ReAct demonstrations) yield peak performance for most models; Rules are ~40% cheaper in tokens (~106M vs ~177M for Gemini) with strong performance and faster graduation. Mixed offers no clear advantage.
Counterintuitive finding: the worse the zero-shot model, the larger the gain — FORGE acts as a *variance-reduction* / capability-gap-bridging mechanism rather than making strong models stronger.
An ablation without graduation sometimes performs better (Grok, Qwen), suggesting graduation's main value is compute savings rather than accuracy.

Why it matters

FORGE demonstrates that LLM agents can acquire reliable long-horizon strategies through population selection in a discrete symbolic (natural-language) space — evolution-like learning with zero gradient updates. If a mid-tier model with a good evolution infrastructure approaches top-tier performance, AI capability gaps may increasingly live in *evolution infrastructure* rather than base model parameters.

Limitations

All experiments use only CAGE-2's B-line attacker and 30-step horizon; cross-environment transfer and cross-model memory portability (e.g., Gemini-evolved memory reused by Llama) remain untested.

References

Bogdanov, I., Lung, C.-H., Kunz, T., Gao, J., Taylor, A., & Zaman, M. (2026). FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast. arXiv:2605.16233 [cs.AI].
Shinn, N., et al. (2023). Reflexion: Language agents with verbal reinforcement learning. NeurIPS 36.
Yao, S., et al. (2023). ReAct: Synergizing reasoning and acting in language models. ICLR.
Jaderberg, M., et al. (2017). Population based training of neural networks. arXiv:1711.09846.
Standen, M., et al. (2021). CybORG: A gym for the development of autonomous cyber agents. arXiv:2108.09118.