Overview
Research area: NLP Authors: Jundong Xu, Qingchuan Li, Jiaying Wu Published: 2025-06-13 arXiv: 2506.10671
Summary
Large language model (LLM) agents have achieved strong performance on a wide range of benchmarks, yet most evaluations assume static environments. In contrast, real-world deployment is inherently dynamic, requiring agents to continually align their knowledge, skills, and behavior with changing environments and updated task conditions.
To address this gap, the authors introduce EvoArena, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains. They further propose EvoMem, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory.
Key findings
- Current agents struggle on EvoArena, achieving only 39.6% average accuracy in evolving terminal, software, and social preference domains.
- EvoMem consistently improves performance: an average gain of 1.5% on EvoArena, 6.1% on GAIA, and 4.8% on LoCoMo standard benchmarks.
- On chained task settings—where success requires completing a series of related evolving subtasks—EvoMem further boosts EvoArena accuracy by 3.7%.
- Mechanistic analysis shows EvoMem improves evidence capture in memory, indicating better preservation of complete evolving environment states.
- Paper: https://arxiv.org/abs/2506.10671
The results underscore the importance of modeling evolution in both evaluation and memory design for reliable agent deployment.
Links
*Auto-collected on 2026-06-13*