How Supermemory's ASMR Hits 99% on LongMemEval: A Multi-Agent Approach to AI Memory
> Imagine a friend who reintroduces themselves every time you meet — no memory of yesterday's conversation, your preferences, or your dietary restrictions. That's the state of most AI today.
---
The "Goldfish Memory" Problem
LLMs are inherently stateless processors: they receive text, generate text, and forget. To compensate, engineers built RAG (Retrieval-Augmented Generation): chunk conversation history, embed it into vectors, store it in a vector database, and retrieve the most semantically similar chunks at query time.
But vector similarity is fuzzy. It struggles with temporal ordering, information updates, and fact overrides. Example:
- User says "I'm going to Shanghai next Wednesday," then later "Actually, changed to Thursday."
- Vector search retrieves both statements, and the AI has to guess which is current.
- Result: confusion, contradiction, wrong answers.
- 115k-token conversation histories (a novella's length)
- Fragmented events scattered across sessions
- Contradictory information (changed minds, updated facts)
- Temporal reasoning questions ("Was that meeting before or after my move?")
- Extractor: explicit facts ("I like coffee", "I work at Google")
- Profiler: user profile (preferences, habits, traits)
- Chronologist: timeline of events
- Fact Finder: directly relevant facts
- Context Miner: implicit context and associations
- Timeline Reconstructor: event ordering
- Strategy 1 — 8 parallel experts (counting, temporal, context-digging, etc.): if any reasoning path answers correctly, it counts. Accuracy: 98.6%
- Strategy 2 — 12 voting agents aggregated by GPT-4o-mini, weighing domain trust and answer consistency. Accuracy: 97.2%
- documentDate: when the conversation happened
- eventDate: when the described event happens
- updates: contradictions/corrections ("favorite color is now green" updates "blue")
- extends: additions (adding "position: engineer" to "works at Google")
- derives: inferred facts ("works at Google" + "lives in Singapore" → "works at Google's Singapore office")
- Very low latency: ~50ms per API call to generate a user profile
- Embeddability: works in any system, even robot hardware
- Simplicity: no embedding models, vector indexes, or chunking strategies to manage
- For the agent ecosystem: billions of AI agents today can't build long-term personalized relationships. A memory-capable assistant remembers your allergies from three months ago, tracks your evolving plans, and knows which "that meeting" refers to. Memory turns AI from novelty into utility.
- Methodological lesson: sometimes a smarter algorithm beats a smarter index. Instead of optimizing similarity math, make the searcher smarter — a shift from "retrieval systems" to "understanding systems."
- Open source: Supermemory has committed to open-sourcing ASMR's full code in early April.
- Supermemory research: https://supermemory.ai/research/
- LongMemEval paper: https://arxiv.org/abs/2410.10813
- Supermemory GitHub: https://github.com/supermemoryai/supermemory
- Dhravya Shah's blog: https://supermemory.ai/blog/
With hundreds of thousands of tokens and facts scattered across dozens of sessions, traditional retrieval becomes needle-in-a-haystack search — and often pulls up rusty junk.
LongMemEval: The Hardest Memory Benchmark
LongMemEval features:
Scoring above 80% is considered top-tier; most systems barely pass. Supermemory's new system, ASMR, scored 99%.
What Is ASMR?
ASMR stands for Agentic Search and Memory Retrieval. Its core idea:
> Instead of making AI "match" similarity, let it understand the information.
Traditional RAG vs. ASMR
Traditional RAG: chunk text → generate embeddings → store in vector DB → similarity search → feed top chunks to the model. Like finding books by title/summary similarity — similar isn't relevant, relevant isn't correct, correct isn't current.
ASMR: 1. Multiple specialized AIs "read" raw conversations 2. Extract structured knowledge points (who, what, when, where, why) 3. Store both original text and extracted memories 4. Dispatch multiple "search agents" that actively reason — reading and understanding, not matching vectors
The Agent Pipeline: 3 + 3 + Many
Layer 1 — Three Observer Agents (running on Gemini 2.0 Flash), processing new conversations in parallel:
They extract structured information along six dimensions: personal info, preferences, events, time data, locations, and implicit context. Crucially, they store structured knowledge points, not vector embeddings.
Layer 2 — Three Search Agents (at query time):
These agents actively read and reason: they determine which information is most current, resolve contradictions, and infer temporal order.
Layer 3 — Parallel variants:
Key Technical Innovations
1. Ditching the Vector Database
Counterintuitive, but vector similarity can't capture precise semantic relations. When "my favorite color is blue" is later overridden by "now it's green," retrieval returns both. ASMR's Observers detect this as an information update and build version history; Fact Finder knows to take the latest version.2. Dual Timestamps
Every memory carries:E.g., "I'm going to Shanghai next week" (document 2024-01-01, event 2024-01-08) vs. "I met Mr. Zhang on my Shanghai business trip" (document 2024-01-10, event 2024-01-08). The Timeline Reconstructor infers the trip followed the plan — same event, ordered phases.
3. Relation Versioning
Three relation types between memories:This creates a dynamically evolving knowledge graph, not a static text collection.
4. Pure In-Memory Operation
No external vector database means:LongMemEval Results
| Category | Description | ASMR Performance | |-----|------|---------| | Single-Session User | User info in one session | Near-perfect | | Single-Session Assistant | What the AI said | Near-perfect | | Single-Session Preference | Implicit user preferences | Significant improvement | | Knowledge Update | Recalling updated facts | Clear advantage | | Temporal Reasoning | Event ordering | 76.69% (historical weak spot) | | Multi-Session | Cross-session integration | 71.43% (historical weak spot) |
Temporal Reasoning and Multi-Session have always been the weakest categories for vector-based methods — similarity can't express "before/after," and cross-session facts often lack semantic overlap. ASMR's explicit timeline management and active reasoning break through here.
Why It Matters
Supermemory's bold claim: "Agent memory might be completely solved now."
Limitations
1. Cost: every query involves multiple LLM calls — far more expensive than a single vector lookup. Economics at scale remain unproven. 2. Experimental status: Supermemory explicitly calls ASMR a highly experimental agentic pipeline, not yet in their production engine. The 99% came under ideal conditions. 3. Generality: LongMemEval is synthetic; real-world memory scenarios may be messier. Performance in verticals (medicine, law, finance) is untested. 4. Not the end of vector DBs: for massive document retrieval and similarity recommendations, vector search remains effective. ASMR addresses a specific but critical problem domain: memory requiring deep understanding and reasoning.
Conclusion
Human identity rests on memory. For AI to become a true companion, it needs a past to understand the present and history to predict the future. ASMR's 99% isn't just a number — it's a signal that the era of "reintroducing yourself to your AI every session" may finally be ending.