Introduction: An Agent's 100 Failures
Imagine a network defense game on CybORG CAGE-2: you are a blue-team defender protecting a 13-host enterprise network against a red-team attacker following a standard kill chain (reconnaissance → intrusion → lateral movement → privilege escalation → objective capture). Each step, you can monitor or analyze a host, remove suspicious processes, restore a machine, or deploy decoys. You face:
1. Long horizon: 30 steps per episode 2. Partial observability: only noisy host-level signals 3. Scalar reward: no natural-language feedback, just a score 4. Randomness: attacker behavior varies per episode
Zero-shot, frontier LLMs fail catastrophically:
- Gemini-2.5-Flash-Lite: -189.6 (worse than random, -154)
- Grok-4-Fast: -58.4
- Llama-4-Maverick: -113.1
- Qwen3-235B: -103.3
- Rules: conditional heuristics (e.g., "on detecting lateral movement: isolate path + analyze adjacent hosts")
- Examples: structured few-shot demonstrations of full Thought→Tool→Observation→Answer loops
- Mixed: both
- Bogdanov, I., Lung, C.-H., Kunz, T., Gao, J., Taylor, A., & Zaman, M. (2026). FORGE: Self-Evolving Agent Memory With No Weight Updates via Population Broadcast. arXiv:2605.16233 [cs.AI].
- Shinn, N., et al. (2023). Reflexion: Language agents with verbal reinforcement learning. NeurIPS 36.
- Yao, S., et al. (2023). ReAct: Synergizing reasoning and acting in language models. ICLR.
- Jaderberg, M., et al. (2017). Population based training of neural networks. arXiv:1711.09846.
- Standen, M., et al. (2021). CybORG: A gym for the development of autonomous cyber agents. arXiv:2108.09118.
Doing nothing scores -218; top RL algorithms reach -3.47.
FORGE (Bogdanov et al., arXiv:2605.16233) asks: can an agent learn from failures without any weight updates — purely through natural language?
The Three-Layer Architecture
Inner loop: anatomy of failure
The agent uses a hierarchical ReAct architecture: a Planner calls an Analyst (interprets host observations) and an ActionChooser (ranks and justifies actions). Each sub-agent maintains persistent user-provided memory plus dynamic memory built during training. Three representation modes:
When a step's reward falls below threshold τ = -1.1 (selected from the reward's natural clusters: legal Restore = -1.0; minor failures = -1.1/-1.2; medium = -2.0/-3.2; severe = -11/-14), the episode aborts. A learning agent (Reflector for Rules, Exemplifier for Examples) generates a new memory artifact from the trajectory, which is appended to dynamic memory; up to k_A = 3 retries per episode, followed by a checkpoint evaluation.
Outer loop: champion broadcast
N = 10 independent agent instances run for S = 6 phases. After each phase, the best checkpoint scorer becomes the Champion, and its entire memory is destructively broadcast: all other instances discard their memory and adopt the champion's. This mirrors Population-Based Training (PBT), but transferred from continuous weight space to discrete text space: copy the best memory, then let independent exploration act as mutation. Merging conflicting rule-sets in natural language is unreliable, so cloning the winner is preferred.
Graduation
Any instance scoring above θ = -15 (roughly 10x better than random) graduates: its memory is frozen and it no longer participates in broadcasts — protecting good solutions from being overwritten, while saving compute as active instances shrink.
Results
Overall performance (best representation per model after FORGE):
| Model | Zero-shot | FORGE | Improvement | |---|---|---|---| | Gemini (Examples) | -189.6 ± 53.9 | -24.5 ± 21.1 | 7.7x | | Grok (Rules) | -58.4 ± 55.2 | -33.7 ± 26.1 | 1.7x | | Llama (Examples) | -113.1 ± 81.5 | -28.3 ± 15.7 | 4.0x | | Qwen (Examples) | -103.3 ± 87.3 | -24.3 ± 35.8 | 4.3x |
Gemini went from worse-than-random to near top-RL level (best single checkpoint: -3.60 vs. DRL's -3.47).
vs. Reflexion: FORGE outperforms single-thread Reflexion on all 12 model×representation combinations, by 29%-72%. The bottleneck of prompt-only adaptation isn't reflection quality but the absence of selection pressure to spread rare good discoveries across a population.
Memory representations: Examples give the best final performance (richest context); Rules are ~40% cheaper in tokens (~106M vs. ~177M for Gemini repeats) with faster convergence and higher graduation rates; Mixed shows no clear advantage.
Weaker models gain more: the worse the zero-shot score, the larger the improvement multiplier. FORGE functions more like variance compression than capability amplification — a potential "capability gap bridger" letting mid-tier models approach top-tier performance.
Graduation ablation: a no-graduation variant sometimes scored better (Grok, Qwen), suggesting graduation's main value is computational savings rather than pure performance.
An Ant-Colony Analogy
One ant alone in the dark repeats its mistakes and may record wrong lessons. Ten ants exploring in parallel, with a "messenger ant" periodically copying the farthest-advancing ant's full experience to everyone else, converts isolated luck into collective knowledge. Discarded exploration isn't wasted because merging two written "defense manuals" is far harder than adopting the better one. Graduation is a retirement system: once an ant finds a solid structural segment, it's preserved from being overwritten.
Why It Matters
FORGE demonstrates that LLM agents can acquire reliable long-horizon strategies through population-level selection in discrete symbolic space — closer to biological evolution than gradient descent — with zero weight updates. If a mid-tier model plus a good evolutionary system can rival a stronger model with naive learning, the future capability gap may lie in "evolution infrastructure" (better reflection, broadcast, memory representation) rather than raw model scale.
Limitations: experiments cover only CAGE-2's B-line attacker with a 30-step horizon; transfer to other environments and cross-model memory transfer remain untested.