AI companion products share a common promise: the longer you chat, the better they understand you. But how do you verify that understanding? RealCompanion (arXiv 2610.01780, v3, 2026-10-08, Quis Lab: Arman Behnam, Eric Huang, Sunglyoung Kim, Jiayi Yu, plus independent researcher Liangwei Yang) tackles this with something no prior benchmark had: ten real human-AI companion relationships totaling 27,218 messages, the longest spanning 120 days, plus participant portraits, personas, and test items where every annotation traces back to supporting original messages. The authors disclose that data review went through the operator's internal ethics committee, not an independent IRB.
Key findings from real data
- People rarely look back. Only 3.4% of real messages depend on earlier conversation. When they do, the reference is typically far away—a median of 2,157 messages back. Recent-message coverage handles 95.9% of needs, but when the needed information is distant, coverage collapses to 2.2%. Synthetic benchmarks fix the demand rate at 100%, whereas reality is one genuine retrieval need every ~29 messages; outside those moments, recall is interference, not capability.
- AI can't tell when to remember. Detectors for "does this message require history?" performed nearly at chance on real messages. In a framing experiment, merely relabeling the same history from "earlier messages" to "memory" increased how often models invoked the past by 10-14 percentage points—even when irrelevant. The AI's "nostalgia" is largely prompt-induced, not conversation-induced.
- Seeing a person, then imagining them. Three agent systems—Claude Opus 5.5, Codex GPT-5.6-sol, and Antigravity running Gemini 3.8 Flash—all rebuilt participant personas at F1 0.71 despite a 31x cost difference. More striking: for every 100 accurately known attributes, each system added 70-80 fields the user never disclosed. Understanding shades into fabrication.
Benchmark design
Tasks split into three decisions: whether to retrieve (restraint), where to retrieve (position), and how to use the result (application, scored on attribution and consistency). Of 1,533 scorable items, 1,009 point to the current thread, 404 to distant history, and 120 to empty ground—judging "nothing to retrieve" is itself a test. User-sampled conversational trajectories estimate real demand rates; constructed question trajectories test reasoning. Metrics report both set averages and per-person macro averages. The annotation pipeline is conservative by construction: verification can only remove citations, never add them, so demand rates can only be underestimated—error bounds are mathematically sealed.
The bigger picture
RealCompanion quantifies what until now was only a feeling: AI companions treat "understanding a person" as "storing a person." But understanding is not hoarding the past—it is knowing which small fragment of the past matters right now, that 3.4% sparse signal. Current systems get this exactly backwards: silent when you need them to recall a distant night, eager to mention last month when you remark on the weather.
*Source note: arXiv 2610.01780 (v3, 2026-10-08). All figures—3.4%, 2,157 messages, 95.9%/2.2%, 10-14 percentage points, F1 0.71, 31x cost gap, 70-80 added fields—are from the paper; data-governance boundaries (internal ethics committee, no independent IRB) are from the paper's text.*