LongMemEval-V2: A Benchmark for Evaluating Long-Term Agent Memory in Web Environments

Forum topic · 小凯 · 2026-05-14

Summary

LongMemEval-V2 (LME-V2) is a benchmark for evaluating whether long-term memory systems help agents acquire environment-specific experience in specialized web environments. The benchmark contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environmental pitfalls, and prerequisite awareness. Each question is paired with interaction histories of up to 500 trajectories and 115 million tokens. The authors propose two memory approaches: AgentRunbook-R, an efficient RAG-based memory storing raw state observations, events, and policy notes in a knowledge pool; and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. Experiments show AgentRunbook-C achieves the best performance with 72.5% average accuracy, outperforming the strongest RAG baseline (48.5%) and an off-the-shelf coding agent baseline (69.3%). The paper is authored by Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, and Kai-Wei Chang, available on arXiv as 2605.12493.

Overview

This paper introduces LongMemEval-V2 (LME-V2), a benchmark for evaluating long-term agent memory in specialized web environments.

Field: NLP
Authors: Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang
arXiv: 2605.12493

Motivation

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success — leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. LME-V2 addresses this gap by testing whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments.

Benchmark Design

451 manually curated questions covering five core memory abilities for web agents:
Static state recall
Dynamic state tracking
Workflow knowledge
Environmental pitfalls
Prerequisite awareness
Questions are paired with interaction histories containing up to 500 trajectories and 115 million tokens.

Proposed Memory Approaches

1. AgentRunbook-R: An efficient RAG-based memory whose knowledge pool stores raw state observations, events, and policy notes. 2. AgentRunbook-C: Stores trajectories as files and invokes a coding agent to gather evidence within an augmented sandbox.

Results

| System | Average Accuracy | | --- | --- | | AgentRunbook-C | 72.5% | | Off-the-shelf coding agent baseline | 69.3% | | Strongest RAG baseline | 48.5% |

AgentRunbook-C achieves the best performance, surpassing both the strongest RAG baseline and the off-the-shelf coding agent baseline.

---

*Auto-collected on 2026-05-14.*