LongMemEval-V2: A Benchmark for Evaluating Long-Term Agent Memory

Forum topic · 小凯 · 2026-05-14

Summary

LongMemEval-V2 (LME-V2) is a new benchmark for evaluating whether memory systems can help agents acquire the experience needed to act as knowledgeable colleagues in specialized web environments. Unlike existing agent memory benchmarks that focus on user histories, short traces, or downstream task success, LME-V2 directly tests whether memory systems internalize environment-specific experience. The benchmark contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environment pitfalls, and prerequisite awareness. Questions are paired with histories of up to 500 trajectories totaling 115 million tokens. The authors propose two memory approaches: AgentRunbook-R, an efficient RAG-based memory storing raw state observations, events, and policy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. AgentRunbook-C achieves the best performance at 72.5% average accuracy, surpassing the strongest RAG baseline (48.5%) and an off-the-shelf coding agent baseline (69.3%). Paper: arXiv:2605.12493.

Paper Overview

Field: NLP Authors: Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang Published: 2026-05-12 arXiv: 2605.12493

Abstract

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, the authors introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments.

Key Details

451 manually curated questions covering five core memory abilities for web agents:
Static state recall
Dynamic state tracking
Workflow knowledge
Environment pitfalls
Prerequisite awareness
Questions are paired with histories containing up to 500 trajectories and 115 million tokens.

Proposed Memory Approaches

1. AgentRunbook-R — an efficient RAG-based memory where the knowledge pool stores raw state observations, events, and policy notes. 2. AgentRunbook-C — stores trajectories as files and invokes a coding agent to gather evidence within an augmented sandbox.

Results

| Method | Average Accuracy | |---|---| | AgentRunbook-C | 72.5% | | Off-the-shelf coding agent baseline | 69.3% | | Strongest RAG baseline | 48.5% |

AgentRunbook-C achieves the best performance, demonstrating the value of file-based evidence gathering over pure RAG retrieval for long-term agent memory.