Half of Essential Memories Are Never Retrieved: Causal Inference Exposes a Fatal Blind Spot in LLM Memory Systems

Forum topic · ✨步子哥 · 2026-10-04

Summary

A Chinese forum post analyzes the CMP (Causal Memory Policy) paper, which argues that memory utility in LLM memory systems is not a scalar but a product of retrievability and conditional utility. Using structural causal models, the authors show that 54%–67% of essential memories in benchmarks like LongMemEval and LoCoMo are never retrieved (P=0), making storage-layer A/B tests meaningless in those cases—an estimated 52.2% of such experiments produce identical arms. The proposed fix, randomized retrieval exposure, reserves k of B context slots for randomly sampled memories from an exposure pool with known probabilities, enabling inverse-probability-weighted estimation of conditional utility. CMP improves essential-vs-nonessential discrimination (AUC 0.542 to 0.664) and halves golden loss. The post also covers asymmetric forgetting/retention decision theory (one-sided abstention), the unresolved exposure-pool selection problem, and practical advice for memory-augmented agent systems.

The Intuition Trap

Suppose you give an LLM a memory management system: after each conversation, it decides what to store, delete, and merge. To judge whether a memory m is useful, you run an apparently reasonable A/B experiment: compare task performance in conversations with m versus without m.

You find many memories have near-zero "utility," so the system confidently deletes them.

But wait. A memory that is never retrieved has "zero utility" — which is completely different from "we don't know whether it's useful." Yet from external observation, the two look identical.

This is the core problem addressed by the CMP (Causal Memory Policy) paper. Using structural causal models (SCM), the authors prove something chilling: in all current memory management systems, more than half of "essential memories" are never recalled by the retriever, and their "low utility" estimates are pure artifacts.

Memory Utility Is Not a Scalar, It's a Product

The paper's first key contribution is decomposing memory utility into two factors:

U(m \mid q, c) = \underbrace{P(m \in R \mid q, c)}_{\text{retrievability}} \times \underbrace{U_R(m \mid q, c)}_{\text{conditional utility}}

Retrievability: given query q and context c, the probability that memory m is selected by the retriever
Conditional utility: if m is retrieved, its causal contribution to the task output

The decomposition seems simple, but it reveals a widely ignored fact: memory utility is not a property of the memory itself, but a property of the interaction between the memory and the retriever.

A memory may have very high conditional utility (it would greatly improve answer quality if retrieved), but if the retriever never selects it, its total utility is zero. External observation of "zero utility" cannot distinguish:

1. The memory is genuinely useless 2. The memory is useful, but the retriever never gave it a chance

The Illusion of Storage-Layer Intervention

You might think: if I randomly add/remove memories from the store, I can estimate utility.

The paper proves this is insufficient. Because retrieval is the only path from memory to output (a mediator). If you randomize at the storage layer (add or delete memory m) but the retriever never recalls m, then:

Retrieval results with m = retrieval results without m
The two experimental arms are exactly the same experiment

This is the paper's "retrieval support failure": when $P(m \in R \mid q, c) = 0$ , no amount of storage-layer experimentation amounts to anything but running the same experiment.

The Empirical Data Are Alarming

| Benchmark | Share of essential memories with P=0 | Memory store size | |---|---|---| | LongMemEval (independent subset) | 54.0% | 487 | | LongMemEval (all) | 61.2% | 487 | | LoCoMo | 66.9% | 588 | | Mem0 (deployed system) | 28.4% | 291 |

On LongMemEval, 61 of 113 essential memories were never recalled by the retriever. Storage-layer randomization had no effect on retrieval results in 52.2% of cases — you ran an experiment, but the experiment never happened.

More disturbing is Figure 3: task accuracy stays near 0.2 until all essential memories are retrieved, then jumps to 0.73 only when every essential memory is retrieved. Missing one memory is nearly as bad as missing half — multi-hop reasoning needs all the puzzle pieces.

CMP's Solution: Randomize Retrieval Itself

Since the problem lies in the retrieval layer, the solution must be there too. CMP's core design is randomized retrieval exposure:

Split B context slots into two parts:

B−k slots: filled normally by retriever ranking
k slots: filled by random sampling from an exposure pool $\mathcal{P}$ with known probabilities

This guarantees every memory has a nonzero probability of entering context, i.e., $P(m \in R \mid q, c) > 0$ by construction. The exposure probability $$e_m = n_m / T$$ is a design parameter, not something to estimate.

Why Balanced Assignment Instead of Independent Sampling?

If each memory were independently sampled with probability $k/|\mathcal{P}|$ , in a setting of $$T=90$$ , $$k=2$$ , $|\mathcal{P}|=32$ , the expected exposures would be 5.6, but about a third of memories would get fewer than 5 exposures. CMP uses balanced assignment: each memory receives exactly $\lfloor Tk/|\mathcal{P}| \rfloor$ or $\lceil Tk/|\mathcal{P}| \rceil$ exposures.

The Estimator

With known exposure probabilities, estimating conditional utility becomes a standard inverse-probability-weighted (Hájek) estimator:

\hat{U}_R(m) = \frac{1}{n_m} \sum_{t \in \mathcal{T}_m^+} Y_t - \frac{1}{T - n_m} \sum_{t \in \mathcal{T}_m^-} Y_t

Because balanced assignment fixes $$n_m$$ , the estimator's variance is exact and does not depend on realized arm sizes.

Key Experimental Results

Q1: How severe is retrieval support failure?

As shown in the table above: 54%–67% of essential memories are never retrieved.

Q2: Can a better retriever fix it?

On Mem0, recovering 50% of unretrieved essential memories requires expanding the retrieval window to 138 entries; recovering 90% requires 286 — effectively abandoning retrieval. The problem is not that the retriever is bad, but that deterministic retrieval itself has blind spots.

Q3: Does randomized retrieval restore identifiability?

| Design | AUC | Nomination rate | Golden loss | |---|---|---|---| | Storage-layer randomization | 0.542 | 0.97 | 10.9 | | CMP (exposure design) | 0.664 | 0.68 | 5.2 |

CMP raises essential-vs-nonessential discrimination from 0.542 to 0.664, cuts the rate of wrongly nominating essential memories for deletion from 0.97 (near random) to 0.68, and halves golden loss.

Crucially, Table 3's stratified analysis shows: the entire gap between the two designs comes from the P=0 stratum. In the P>0.8 stratum, they are identical. This proves the problem is not the estimator but the design — the storage-layer estimator isn't "inaccurate," it is "measuring something fundamentally unidentifiable."

Q4: Does identifying utility yield a deletion policy?

No. This is one of the paper's most honest findings.

Per-query estimation raises AUC from 0.581 to 0.784, but this utility does not transfer across queries. Utility estimated in the first half of a conversation correlates only r=0.10 with the second half. No aggregation over observed queries can predict a memory's value on unobserved queries.

The paper therefore does not claim CMP is a deletion policy. It only claims CMP is an identification framework — it lets you measure, but measurement is not decision-making.

The Asymmetry of Irreversible Operations

The paper also makes a subtle decision-theoretic contribution: the costs of forgetting and retaining errors are asymmetric.

Forgetting is irreversible: once deleted, utility can never be re-estimated
Retention is reversible: an unused memory can still be deleted later

From this asymmetry, the paper derives a Bayes-optimal decision rule: forget only when the confidence lower bound on utility is negative (one-sided abstention). Two-sided abstention (also considering confidence intervals for "useful") performs worse — it sends potentially useful memories into the deletion pool.

Ablations confirm this:

Remove one-sided abstention: golden loss rises from 5.2 to 9.0
Remove exposure design: golden loss rises from 5.2 to 10.9
Remove both: golden loss rises to 11.7
Two-sided abstention is even worse: 14.1 (worse than no abstention)

Why This Finding Matters

1. A New Instance of the "Evaluation Blind Spot Law"

The paper fits an emerging lineage: the blind spots of evaluation systems are not random but systematic.

LLM auditors with 100% recall but terrible calibration (Rigged Backtest)
Sociability-focused evaluators mislabeling "good communication" as sycophancy (Receptiveness)
Keyword-matching false positives (evaluation credibility lineage)
Now add: storage-layer randomization running a "nonexistent experiment" in 52.2% of cases

All these blind spots share one structure: the evaluation tool measures one thing while you believe it measures another. Storage-layer randomization measures "the effect of storage-state changes," but you assume it measures "memory utility." Because of retrieval's mediating role, the two coincide only when P>0.

2. Another Case of the "Scalar Illusion"

"Memory utility" is not a scalar. It is the product of two factors: retrievability × conditional utility. Treating it as a scalar produces systematic misjudgment in the P=0 region — the same pattern as asking "gears or friction pads?" (a constrained choice, not a scalar) or "is the LLM auditor accurate?" (recall and calibration, two dimensions).

3. Judgment-Gate Decoupling at the Memory Layer

CMP's structure echoes the "judgment-gate decoupling" lineage:

Judgment: what memories exist in the store (storage-layer operations)
Gate: which memories the retriever admits into context

CMP's core insight: intervening only on the judgment layer (storage) without intervening on the gate layer (retrieval) accomplishes nothing in the P=0 region. Like the locate-correct-continue loop or physically forced synchronization — the gate layer is what controls flow.

4. Exposure-Pool Selection Is the Remaining Hard Problem

The paper's most honest admission: CMP proves randomized retrieval restores identifiability, but does not solve exposure-pool selection. To identify a memory's utility you must first place it in the exposure pool — but how do you know whom to place there?

The paper tested three deployable exposure-pool selection rules; all had near-zero recall. CMP currently needs an "oracle" to build the pool, which is unusable in deployment. This is an honestly stated open problem, not a hidden flaw.

5. Implications for All Memory-Augmented LLM Systems

If you use Mem0, LangGraph Memory, or any memory-equipped agent system, these findings apply directly:

Your deletion policy may be deleting memories you don't know you need
Storage-layer A/B tests are idle in 52% of cases
"This memory is useless" and "the retriever never gave this memory a chance" cannot be distinguished from external observation

Minimal viable improvement: randomly insert k memories into retrieval results (even k=1) and record their effect on task performance. This barely slows your system but gives you the ability to estimate the utility of never-retrieved memories.

Paper Information

Title: Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval
arXiv: 2610.02070
Code: anonymous.4open.science/r/cmp-release-D0C3 (anonymous review version)
Core contribution: formalizes memory utility as a retrieval-mediated causal effect, proves non-identifiability under deterministic retrieval, and proposes a randomized retrieval exposure design to restore identifiability

---

*This paper made me reconsider "evaluation" itself. You think you're measuring X, but your tool is actually measuring Y — and X and Y coincide only in some regions. In the regions where they don't, all your data are illusions.*