English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

MemPABench: Benchmarking Personal Agent Memory for Interaction Preferences

Forum topic · 小凯 · 2026-10-24

Summary

MemPABench is a new benchmark for evaluating memory in personal AI agents, focusing not on factual recall but on how well agents learn and follow user interaction preferences. Built as an interactive on-policy environment, it presents long-horizon tasks where agents must decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts, providing feedback over 99 sessions spanning four months, with five event-driven preference changes per user to test memory updating. The authors evaluate four memory systems across four LLM backbones on preference selection, delivered behavior, and sensitivity to context and preference change. Findings show current memory offers limited gains: retrieval-based systems (Mem0, MemOS, simple RAG) do not outperform no-memory baselines, while file-based memory improves accuracy but over-applies preferences to irrelevant contexts. Even when the correct preference is available, overriding an agent's default interaction habits remains difficult. The environment and dataset are open-sourced.

Paper Overview

  • Field: NLP
  • Authors: Jialin Li, Zhenhao Chen, Hanjun Luo, Hanan Salam
  • Published: 2026-10-24
  • arXiv: 2610.30485

    • Why This Paper Matters

    Personal intelligence systems increasingly handle complex everyday tasks, but repeated questions, unnecessary confirmations, and unwanted outreach turn capable assistance into interaction burden. Reducing this burden requires agents to remember *how* users want to be assisted — yet existing memory benchmarks focus on factual recall and personalized answers, rarely testing how agents learn and act on interaction preferences.

    MemPABench

    An interactive environment for on-policy evaluation of memory in personal agents:
  • Long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously
  • Four grounded simulated users holding 14 preferences across work and personal contexts
  • Feedback spanning 99 sessions over four months
  • Five event-driven preference changes per user to test memory updates

    • Evaluation Dimensions

    1. Preference selection 2. Delivered behavior 3. Sensitivity to context and preference change

    Key Findings

  • Current memory brings limited gains in preference selection
  • Retrieval-based systems (Mem0, MemOS, simple RAG) do not outperform No Memory
  • File-based memory improves accuracy, but applies preferences beyond relevant contexts
  • Current memory struggles to retrieve interaction preferences and adapt to preference changes
  • Even with the correct preference available, overriding the agent's default interaction habits remains challenging
The environment and dataset are open-sourced (see the arXiv page for the link).

--- *Auto-collected on 2026-10-24*

Tags

#ai-agents#agent-memory#benchmark#llm#nlp#mempabench#interaction-preferences

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636393