One Token per Evidence: How Latent Memory Rewrites RAG's Compression Rules

Forum topic · 小凯 · 2026-06-14

Summary

A National University of Singapore paper proposes Latent Memory, a RAG framework that compresses each retrieved evidence item—text or image—into a single latent token (a 2048- or 4096-dimensional vector) that is used for both retrieval and generation. A small compressor (LLaMA-3.2-1B for text, LLaVA-1.5-7B for multimodal input) encodes evidence via a learnable [MEM] token, trained jointly with three objectives: reconstruction, contrastive retrieval, and KL distillation against a frozen generator (LLaMA-3-8B, Mistral-7B, LLaVA-13B). On WebQA multimodal QA, it raises image QA F1 from 53.0 to 69.4 while cutting token consumption about 23x versus dense retrieval; on text QA (HotpotQA, 2Wiki, MuSiQue) it achieves near-dense-retrieval accuracy with roughly 3x fewer tokens. Latent tokens remain semantically interpretable, supporting text and image reconstruction via CLIP embeddings and unCLIP diffusion. The method suits edge and resource-constrained AI but has limits: lossy compression, dependence on QA-pair training data, and unverified scaling to million-document corpora.

RAG (Retrieval-Augmented Generation) has an awkward truth: it lets LLMs read external material, but every retrieved passage or image must be stuffed verbatim into the context window. Ten evidence passages mean thousands of tokens; add images and you're past ten thousand. Mobile assistants and edge-device AI simply cannot afford this overhead.

A paper from the National University of Singapore proposes a blunt but effective idea: store one latent token per evidence item, whether text or image. Not compressed into short text, not a summary—compressed into a single high-dimensional vector (2048 or 4096 dims) that replaces the raw evidence in both retrieval and generation.

It sounds like black magic, but it works.

1. The Traditional RAG Bottleneck: Retrieval and Generation Live in Different Worlds

Current RAG pipelines are split:

The retrieval stage encodes evidence and queries with an embedding model (e.g., BGE, Qwen-Emb) for similarity search. The generation stage feeds the full retrieved text/images to the LLM. Consequences:

Retrieval and generation happen in two different representation spaces
Each retrieved item costs hundreds to thousands of tokens at generation time
Images are worse—ViT splits one image into 576 visual tokens; five images is ~3,000 tokens

On WebQA multimodal QA, the Full Context baseline stuffs 10,000+ tokens into the generator and it collapses (0 F1). Not poor quality—the generator literally cannot fit it.

Latent Memory's answer: do everything, from retrieval to generation, in one latent space.

2. How Can One Token Hold a Passage or an Image?

It doesn't "hold" it—it encodes it. A small compressor (LLaMA-3.2-1B for text, LLaVA-1.5-7B for multimodal) reads the evidence with a learnable [MEM] token prepended; the last-layer hidden state is the latent token:

evidence\ x_i + [MEM] \to \text{compressor } \theta \to z_i \in \mathbb{R}^{2048}\ (\text{or } 4096)

This token isn't for humans—it must satisfy three tasks simultaneously: reconstruct the original semantics, retrieve (be comparable to queries in the same space), and generate (feeding it to the LLM yields answers as good as with raw evidence). Three losses, jointly trained.

3. Three-Objective Training: Mutual Reinforcement, Not Compromise

\mathcal{L} = 0.5 \cdot \mathcal{L}_{\text{Recon}} + 0.2 \cdot \mathcal{L}_{\text{Contrast}} + 1.0 \cdot \mathcal{L}_{\text{Distill}}

Reconstruction Loss: the token must remember

Text uses autoregressive reconstruction via a lightweight decoder:

\mathcal{L}_{\text{Recon}}^{\text{text}} = -\sum_t \log P_\pi(x_{i,t} \mid x_{i,<t}, z_i)

For images, rebuilding pixels is too expensive, so the paper reconstructs the CLIP image embedding:

\mathcal{L}_{\text{Recon}}^{\text{img}} = \sum_i \lVert v_i - \text{MLP}(z_i) \rVert_2^2

The token only needs to remember "what the image looks like in CLIP space"; reconstruction to pixels goes through a pretrained unCLIP diffusion model—reusing existing generative capability.

Contrastive Loss: the latent space must be searchable

Latent tokens are projected to a 512-dim shared retrieval space, L2-normalized, matched by inner product:

e_{z_i} = \ell_2(\text{LayerNorm}(\text{Linear}(z_i))) \in \mathbb{R}^{512}

InfoNCE-style loss pulls the query toward its matching evidence and away from negatives. Key design: queries are encoded by the same compressor backbone but with a separate LoRA, unifying the space while letting retrieval and compression be optimized independently.

Distillation Loss: generation behavior must not change

The generator (LLaMA-3-8B, Mistral-7B, LLaVA-13B) stays frozen. The same question is answered twice: once with raw evidence (teacher distribution), once with latent tokens (student distribution), and KL divergence between the two is minimized:

\mathcal{L}_{\text{Distill}} = \sum_t \text{KL}\left( P(\cdot \mid a_{<t}^{\text{tea}}, Q, C^+, \phi) \,\|\, P(\cdot \mid a_{<t}^{\text{tea}}, Q, b_{z_1},...,b_{z_k}, \phi) \right)

Here $b_{z_i} = W_g z_i$ projects latents into the generator's dimension (2048→4096 text; 4096→5120 multimodal, with a GELU in between). The distillation weight is the highest (1.0)—making the LLM "believe" it's reading raw evidence is the crux of the method.

4. Retrieval and Generation: A Fully Closed Latent Loop

Building the memory bank: every evidence item yields one latent token; storage drops from MB to KB scale.
Retrieval: query goes through the query LoRA, projects to 512 dims, top-k by inner product.
Generation: top-k latents pass through $$W_g$$ and are injected directly into inputs_embeds. The LLM is fully frozen.

No raw character or pixel ever enters the generator—hence the 3–10x token savings.

5. Results: Token Savings Without Performance Collapse

Text QA (HotpotQA / 2Wiki / MuSiQue)

| Method | OOD avg EM | Tokens | |:---|:---|:---| | Full Context | ~22.0 | 1827 | | BM25 (k=5) | ~22.0 | 209 | | Dense (k=5) | ~23.0 | 208 | | Qwen3-Emb (k=5) | ~24.0 | 219 | | Latent Memory (k=5) | 16.5 | 71 |

Latent Memory's OOD EM trails Qwen3-Emb, but the authors emphasize the efficiency–effectiveness Pareto frontier: ~3x fewer tokens for near-optimal results. With Mistral-7B as generator, Latent Memory (EM 3.2) actually beats Qwen3-Emb-ft (2.9).

Multimodal WebQA (the standout result)

| Method | Image QA F1 | Text QA F1 | Avg tokens | |:---|:---|:---|:---| | Full Context | 0.0 (crashed) | 6.0 | 10013 | | Nemo-Emb (k=5) | 53.0 | 48.6 | 1257 | | Latent Memory (k=5) | 69.4 | 30.7 | 83 |

Image QA F1 jumps 31% while tokens drop ~23x. Compressing an image into one latent token while preserving enough cross-modal semantics shows these tokens encode abstract meaning, not mere retrieval IDs.

1 Token vs. 8 Tokens

Scaling to 8 latent tokens per evidence yields roughly linear gains at modest cost (71→106 tokens):

| k | 1-Token EM | 8-Token EM | Δ | |:---|:---|:---|:---| | 1 | 12.8 | 14.4 | +1.6 | | 2 | 14.4 | 17.3 | +2.9 | | 5 | 16.5 | 19.9 | +3.4 |

6. Interpretability: Not a Black Box

Latent tokens are semantically interpretable information carriers. Text reconstruction achieves cross-entropy around 0.6–0.7; one reconstructed passage preserved a counting detail ("three display windows" in London's Burlington Arcade) that helped answer the question. Image reconstruction via CLIP embedding + unCLIP diffusion is not pixel-perfect but preserves semantic content (storefronts, signs, window layouts).

7. Engineering Details

Only the compressor side is trained—compressor LoRA (r=64, α=128), query LoRA, retrieval MLP, projection $$W_g$$ , decoder LoRA. The generator stays frozen, avoiding catastrophic forgetting and making the method pluggable into any pretrained LLM/VLM.

| Setting | Text | Multimodal | |:---|:---|:---| | Compressor | LLaMA-3.2-1B | LLaVA-1.5-7B | | Batch | 8 × grad-accum 4 | 2 | | Training time | ~20 h | ~30 h | | Negatives | up to 8 text | 4 text + 4 image |

Text and images share one compressor and one latent space—no modality-specific embedding mismatch at retrieval time, something traditional multimodal RAG struggles with.

8. Limitations

1. Compression is lossy—for tasks needing exact quotation (legal, medical), vanilla RAG may be safer; 8 tokens is a compromise. 2. Training needs (question, evidence, answer) triples—extra data cost for unlabeled domains. 3. Reconstruction is imperfect—text CE of 0.6–0.7 implies information loss; image reconstruction is semantic, not pixel-level. 4. 512-dim retrieval space is untested at million-document scale; experiments use QA datasets with thousands of evidence items.

9. Why This Paper Matters

Latent Memory's value is not a single-point breakthrough but that it redefines RAG's compression boundary: with proper training, one high-dimensional latent token can carry enough information that the generator behaves as if reading raw evidence.

This opens doors for on-device AI assistants (hundreds of evidence items in a few hundred tokens), real-time multimodal QA over video streams, and long-term memory as latent token libraries. The paper's title, "One Token per Multimodal Evidence," is the manifesto; the subtitle, "Latent Memory for Resource-Constrained QA," is the use case. It's not chasing SOTA—it answers an engineering question: can RAG still work under resource constraints?

The answer: yes, and quite well.

---

Paper: Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee. "One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA." arXiv:2606.10572.