DolphinBench: A Benchmark Mapping the Pareto Frontier of Agent Memory

Forum topic · 小凯 · 2026-09-23

Summary

DolphinBench is a benchmark that evaluates agent memory systems directly through task completion rather than conversational question answering. Motivated by the limitations of existing memory benchmarks—where questions reveal which facts must be retrieved, and accuracy is the only reported metric—DolphinBench introduces three knowledge-work personas, each with roughly 500k tokens of user message history. Agents are evaluated on 200 tasks per persona that depend on that history. Every task is verified by running an agent both with and without the relevant history, requiring success with history and failure without it. The benchmark also requires reporting total cost and latency alongside accuracy, enabling evaluation of the tradeoff frontier of memory systems. According to the authors, no existing memory benchmark combines all three properties: implicit evaluation via task completion, verified task dependencies, and cost/latency reporting. The dataset and evaluation code are available at https://dolphinbench.ai. The paper (arXiv:2609.24971) is authored by Soumil Rathi, Deshraj Yadav, and Taranjeet Singh.

Overview

Field: NLP Authors: Soumil Rathi, Deshraj Yadav, Taranjeet Singh Released: 2026-09-21 arXiv: 2609.24971

Abstract

Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores.

We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history.

Key Features

Task-based evaluation: Memory is assessed implicitly through agent task completion, not explicit QA.
Verified tasks: All 200 tasks per persona are validated by running an agent both with and without the relevant history—requiring success when the history is available and failure when it is not.
Full-efficiency reporting: Evaluations must report total cost and latency in addition to accuracy, giving a comprehensive picture of memory system tradeoffs.

According to the authors, no existing memory benchmark combines all three of these properties.

Resources

Dataset and evaluation code: https://dolphinbench.ai
Paper: https://arxiv.org/abs/2609.24971

--- *Auto-collected on 2026-09-23.*