LongMemEval-V2: A Benchmark for Evaluating Long-Term Agent Memory in Specialized Web Environments

Forum topic · 小凯 · 2026-05-14

Summary

LongMemEval-V2 (LME-V2) is a benchmark from UCLA researchers (Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang) that evaluates whether memory systems can help web agents internalize environment-specific experience, such as interface affordances, state dynamics, workflows, and recurring failure modes. Unlike prior agent memory benchmarks focused on user histories, short traces, or downstream task success, LME-V2 contains 451 manually curated questions spanning five core memory abilities: static state recall, dynamic state tracking, workflow knowledge, environment pitfalls, and prerequisite awareness. Questions are paired with interaction histories of up to 500 trajectories and 115 million tokens. The authors propose two memory approaches: AgentRunbook-R, an efficient RAG-based memory storing raw state observations, events, and policy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. AgentRunbook-C achieves the best performance at 72.5% average accuracy, surpassing the strongest RAG baseline (48.5%) and an off-the-shelf coding-agent baseline (69.3%). Paper: arXiv:2605.12493.

Overview

Field: NLP
Authors: Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang
Published: 2026-05-12
arXiv: 2605.12493

Key Points

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes.
Existing memory benchmarks mostly focus on user histories, short traces, or downstream task success; they do not directly measure whether memory systems internalize environment-specific experience.
LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents:
Static state recall
Dynamic state tracking
Workflow knowledge
Environment pitfalls
Prerequisite awareness
Questions are paired with interaction histories of up to 500 trajectories and 115 million tokens.

Proposed Memory Methods

1. AgentRunbook-R: an efficient RAG-based memory whose knowledge pool stores raw state observations, events, and policy notes. 2. AgentRunbook-C: stores trajectories as files and invokes a coding agent to gather evidence inside an augmented sandbox.

Results

AgentRunbook-C achieves the best performance with 72.5% average accuracy.
It outperforms the strongest RAG baseline (48.5%) and an off-the-shelf coding-agent baseline (69.3%).

Abstract

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments.

---

*Auto-collected on 2026-05-14.*