MEME: Multi-entity & Evolving Memory Evaluation Benchmark for LLM Agents

Forum topic · 小凯 · 2026-05-14

Summary

MEME is a benchmark for evaluating long-term memory in LLM-based agents operating in persistent environments. Unlike prior benchmarks that test only single-entity updates, MEME defines six tasks covering the full space of multi-entity and evolving memory, including three previously unscored tasks: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). The authors evaluate six memory systems across three memory paradigms on 100 controlled episodes. Under default configurations, all systems collapse on dependency reasoning despite adequate static retrieval performance, achieving only 3% average accuracy on Cascade and 1% on Absence. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. Only a file-based agent using Claude Opus 4.7 as its internal LLM partially bridges the gap, but at roughly 70x the cost, suggesting that current solutions to this problem do not scale practically. Paper: arXiv 2605.12477.

Paper Overview

Research Area: NLP Authors: Seokwon Jung, Alexander Rubinstein, Arnas Uselis, Sangdoo Yun, Seong Joon Oh Published: 2026-05-12 arXiv: 2605.12477

Abstract

LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work:

Cascade and Absence — tasks requiring dependency reasoning
Deletion — reasoning about the post-removal state

Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, the authors find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3%, Absence: 1% in average accuracy) despite adequate static retrieval performance.

Key Findings

All six evaluated memory systems fail at dependency reasoning tasks in their default configurations.
Prompt optimization, deeper retrieval, and reduced filler noise do not close the performance gap.
Most stronger LLMs also fail to improve results.
Only a file-based agent paired with Claude Opus 4.7 as its internal LLM partially bridges the gap — but at roughly 70x the cost.
Conclusion: bridging the gap currently depends on configurations that are impractical to scale.

---

*Auto-collected on 2026-05-14*