MEME: Multi-entity & Evolving Memory Evaluation — A New Benchmark for LLM Agent Memory

Forum topic · 小凯 · 2026-05-14

Summary

MEME is a benchmark from researchers including Seokwon Jung and Seong Joon Oh (arXiv:2605.12477) that evaluates how well LLM-based agents store, update, and reason over information across sessions in persistent environments. Unlike prior benchmarks that test only single-entity updates, MEME defines six tasks spanning the full space of multi-entity and evolving memory, including three previously unscored tasks: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Across 100 controlled episodes and six memory systems spanning three memory paradigms, all systems collapsed on dependency reasoning under default configurations, achieving only 3% average accuracy on Cascade and 1% on Absence despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs failed to close the gap. Only a file-based agent paired with Claude Opus 4.7 partially closed it, at roughly 70x the cost, indicating that current solutions to these memory failures do not scale practically.

Paper Overview

Field: NLP
Authors: Seokwon Jung, Alexander Rubinstein, Arnas Uselis, Sangdoo Yun, Seong Joon Oh
Published: 2026-05-12
arXiv: 2605.12477

Abstract

LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work:

Cascade and Absence — dependency reasoning tasks
Deletion — post-removal state reasoning

Key Findings

Six memory systems spanning three memory paradigms were evaluated on 100 controlled episodes.
All systems collapse on dependency reasoning under default configurations:
Cascade: 3% average accuracy
Absence: 1% average accuracy
This occurs despite adequate static retrieval performance.
The following interventions fail to close the gap:
Prompt optimization
Deeper retrieval
Reduced filler noise
Most stronger LLMs
Only a file-based agent paired with Claude Opus 4.7 as the internal LLM partially closed the gap — at approximately 70x the cost, suggesting that current mitigations are not practically scalable.

---

*Auto-collected on 2026-05-14*