MEME: A Benchmark for Multi-Entity and Evolving Memory in LLM Agents

Forum topic · 小凯 · 2026-05-14

Summary

MEME (Multi-entity & Evolving Memory Evaluation) is a benchmark for assessing how LLM-based agents store, update, and reason over information across sessions in persistent environments. Unlike prior benchmarks limited to single-entity updates, MEME defines six tasks covering the full space spanned by the multi-entity and evolving axes, including three previously unscored tasks: Cascade and Absence (dependency reasoning) and Deletion (state after removal). The authors evaluate six memory systems across three memory paradigms on 100 controlled episodes and find that all systems collapse on dependency reasoning under default configurations, achieving only 3% average accuracy on Cascade and 1% on Absence despite adequate static retrieval performance. Interventions such as prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. Only a file-based agent paired with Claude Opus 4.7 as the internal LLM partially bridges the gap, but at roughly 70 times the cost, indicating that current fixes are impractical to scale. The paper is available on arXiv as 2605.12477.

Paper Overview

Field: NLP Authors: Seokwon Jung, Alexander Rubinstein, Arnas Uselis, Sangdoo Yun, Seong Joon Oh Published: 2026-05-12 arXiv: 2605.12477

Key Points

LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions.
Prior benchmarks evaluate only single-entity updates; MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes.
Three tasks were not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state).
Six memory systems spanning three memory paradigms were evaluated on 100 controlled episodes.
All systems collapse on dependency reasoning under the default configuration — Cascade: 3%, Absence: 1% average accuracy — despite adequate static retrieval performance.
Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap.
Only a file-based agent paired with Claude Opus 4.7 as the internal LLM partially bridges the gap, but at roughly 70x the cost, indicating that current solutions depend on configurations that do not scale practically.

Original Abstract (excerpt)

> LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance...

---

*Auto-collected on 2026-05-14.*