Paper Overview
Research Area: Machine Learning Authors: Subrat Panda Posted: 2026-09-25 arXiv: 2609.21982
Summary
Long-conversation memory benchmarks increasingly test recall and prompted knowledge updates. TWIST is a proposed benchmark suite for a complementary, unmeasured property: intervention quality — whether a deployed memory system acts correctly at belief change points.
Four Evaluation Tracks
1. Unprompted tension detection — identifying contradictions without being told where they occur. 2. Vetting outgoing drafts — reviewing generated responses against the transcript before sending. 3. Answering with current beliefs — respecting belief updates while preserving supersession history of prior states. 4. Governing sensitive recall — managing when sensitive memories should or should not surface.
Anti-Gaming Design
The suite extends the LoCoMo corpus, pairing each detection metric with controls that should *not* be flagged. Hard negatives that merely surface-match true contradictions price false interventions, so no track can be gamed by maximizing flags across all items.
Key Findings (Track B v1.0, human-verified gold set)
- No tested configuration achieved high contradiction recall, high hard-negative specificity, and high attribution simultaneously.
- Flat-RAG baselines: detected 0.76–0.97 of true contradictions, but false-flagged 16–43% of surface-matching safe drafts.
- Coherence-oriented systems: almost never over-flag, but catch only ~42% of true contradictions.
- This recall–specificity trade-off is invisible when evaluation relies on recall scores alone.
Original Abstract
Long-conversation memory benchmarks increasingly test recall and prompted knowledge updates. TWIST is a proposed benchmark suite for a complementary, unmeasured property: intervention quality — whether a deployed memory system acts correctly at belief change points. Four tracks cover unprompted tension detection, vetting outgoing drafts, answering with current beliefs while preserving supersession history, and governing sensitive recall.
--- *Auto-collected on 2026-09-26*