EvoArena Deep Dive: When the Environment Keeps Changing, Is Your Agent's Memory Still 'Overwrite-Saving'?

> Paper: EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments > Authors: Jundong Xu, Qingchuan Li, Jiaying Wu, Yihuai Lan, Shuyue Stella Li, Huichi Zhou, Bowen Jiang, Lei Wang, Jun Wang, Anh Tuan Luu, Caiming Xiong, Hae Won Park, Bryan Hooi, Zhiyuan Hu > Institutions: NUS, SMU, UW, UCL, UPenn, NTU, Recursive, MIT > Paper: arXiv:2606.13681 > Code: open-sourced at https://github.com/Aiden0526/EvoArena > Date: 2026-06-11

TL;DR

EvoArena exposes a critical blind spot in existing agents: environments change, but memory stores only the latest state. The authors built three dynamic evolution benchmarks (terminal workflows, codebases, user preferences) and proposed EvoMem — recording memory evolution history as Git-style patches instead of overwriting. Experiments show an average +3.7% chain-level accuracy gain, especially pronounced under continuous environmental change.

The Core Problem: Why Do Agents 'Forget' Once Deployed?

Existing LLM agent evaluation assumes a static environment. But the real world is not:

• APIs your scripts depend on change interfaces
• The codebase you maintain keeps gaining new versions, features, and bugs
• User preferences shift over time — today's 'like' may become tomorrow's 'dislike'

An agent must answer three questions: 1. What changed? (environment evolution detection) 2. What still works? (retaining old knowledge) 3. How to act under the current version? (adaptation)

But current agents merge all memory into a single latest state, causing what the authors call 'State Collapse' — still-valid old rules get overwritten, and the agent fails entirely when environments roll back or multiple versions coexist.

EvoArena: Three Real Evolution Scenarios

Scenario 1: Terminal-Bench-Evo (terminal workflow evolution)

• Evolution: terminal dependencies, interfaces, paths, validation rules change continuously
• Scale: 89 initial tasks → 352 evolved versions, 441 instances total
• Tests: whether an agent can still execute workflows correctly after interface changes

Scenario 2: SWE-Chain-Evo (codebase evolution)

• Evolution: codebase milestones, new features, bug fixes
• Scale: 50 evolution chains, 493 chain-step instances, 145 unique milestones
• Key design: after evaluating step t, reference milestone updates (not the agent's own patches) form the step t+1 repo state — isolating 'adapting to an evolving repo' from 'early error accumulation'

Scenario 3: PersonaMem-Evo (user preference evolution)

• Scale: 10 persona conversations, 505 questions, 313 preference chains
• Four question types: single-pattern transfer, multi-pattern synthesis, conflict resolution, temporal trajectory

EvoMem: Git-Like Patch Memory — No More Overwriting

Core idea: append-only, never overwrite — like Git commit history. Each patch records six elements: timestamp, pre-update content, post-update content, update rationale, semantic summary, supporting evidence.

• At inference, the latest memory is used by default
• When old-version knowledge is needed, relevant patches are retrieved to restore valid states
• Patches are only created for non-additive updates (modifications, overwrites, reinterpretations); pure additions don't trigger recording

EvoMem instantiates across agent types (Terminus2 terminal agent, OpenHands software-engineering agent, A-Mem conversational memory, Memento-S skill memory) as a pluggable patch layer on top of existing memory systems — no model changes or fine-tuning required.

Experimental Results: How Fragile Are Agents in Dynamic Environments?

Baseline chain-level accuracy is dire

| Benchmark | Step-level | Chain-level | |---|---|---| | Terminal-Bench-Evo | 43.6% | 21.5% | | SWE-Chain-Evo | 27.9% | 10.0% | | PersonaMem-Evo | 47.3% | 40.0% |

Chain-level accuracy means 'correct end-to-end'. 10–40% shows agents can barely maintain long-horizon reliability under continuous change: one wrong step, everything after fails.

EvoMem gains

| Benchmark | Step-level | Chain-level | Note | |---|---|---|---| | Terminal-Bench-Evo | +2.4% | +6.1% | chain-level gain is 2.5x step-level | | SWE-Chain-Evo | +0.4% | +2.1% | hardest setting, still improves | | PersonaMem-Evo | +1.7% | +3.2% | temporal-trajectory questions benefit most | | Average | +1.5% | +3.7% | chain > step |

Best case: Terminal-Bench-Evo + GPT-5.5 goes from 31.8% to 45.5% chain-level accuracy (+13.7 points).

EvoMem also helps on non-evolution benchmarks: GAIA (Memento-S) 65.8% → 72.3% (+6.5%); LoCoMo (A-Mem) 39.7% → 43.0% (+3.3%).

Ablations: What's Actually Working?

1. Patches must be operationalized to help: on Terminal-Bench-Evo, when retrieved patches are adopted, accuracy reaches 88.9% vs 49.4% when retrieved but not adopted — the value lies in operationalizing retrieved transition information, not just adding context. 2. SWE chain-level failures drop: Pass-to-Pass failure rate falls from 7.14% → 3.33% for Kimi-K2.6 and 9.01% → 6.73% for Qwen3.6-27B — EvoMem helps agents preserve historical behavioral constraints while implementing new features. 3. PersonaMem splits by question type: multi-pattern synthesis (+5.2%) and temporal trajectory (+5.2%) improve most — exactly where state collapse hits hardest; conflict resolution (-0.9%) and single-pattern transfer (-1.8%) slightly decline.

Accuracy–Efficiency Trade-off: More Tokens ≠ Better

• PersonaMem-Evo: GPT-5.5 scores 40.3% but burns 69.2M tokens (highest); Gemma-4-31B scores best (52.6%) with 27.1M tokens; Kimi-K2.6 is second-best (51.5%) with the lowest consumption (24.5M)
• Terminal-Bench-Evo: GPT-5.5 reaches 62.8% but consumes 505.0M tokens, while Gemini-3.1 Pro / GLM-5.1 hit 53.8%/51.8% at ~80M tokens (vs. 203.6M average)

Conclusion: larger token budgets don't guarantee better performance; EvoMem lets mid-size models exploit memory history efficiently.

Why This Matters

1. State collapse is an implicit bug: memory systems like Mem0, A-Mem, and LangGraph default to 'latest = best', but in reality API v1 interfaces may return in v3, last year's hated feature may become essential, and old codebase constraints may still hold. Merging to a single state loses all context. 2. Chain-level accuracy is the real metric: production cares about 'how long can it run without error', not single-step scores. 3. Lightweight, plug-and-play: EvoMem adds a patch layer on existing memory systems — no retraining. 4. Cross-domain validation: terminal workflows, codebases, and user preferences all benefit, suggesting state collapse is a general problem and patch history a general solution.

Limitations

• Limited evolution scope (no multimodal/physical environment change)
• Patch creation overhead may bottleneck at very high-frequency updates
• Retrieval errors can inject noise; retrieval precision directly affects recovery
• Storage bloat for long-running agents needs compression/archival strategies
• RAG/vector-DB integration untested

One-Line Summary (Again)

EvoArena tells us: an agent isn't fragile because it can't remember — it's because it only remembers 'now', forgetting 'what was' and 'why it changed'. Replacing overwrite-saving with Git-style patch history lets agents know what changed, what still works, and how to act under the current version.

> "Memory is not a snapshot; it is an evolution history."

Reference: Jundong Xu et al. "EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments." arXiv:2606.13681, 2026.