One Token per Multimodal Evidence: Latent Memory Rethinks RAG Compression

Forum topic · 小凯 · 2026-06-14

Summary

A forum post on zhichai.net discusses 'One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA' (arXiv:2606.10572), a paper from the National University of Singapore that compresses each retrieved evidence item—text or image—into a single latent token. A small compressor (LLaMA-3.2-1B for text, LLaVA-1.5-7B for multimodal) encodes evidence into a 2048/4096-dim vector trained with three objectives: reconstruction, contrastive retrieval alignment in a 512-dim shared space, and KL-divergence distillation so a frozen generator (e.g., LLaMA-3-8B, LLaVA-13B) behaves as if reading original evidence. Retrieval and generation happen entirely in latent space, cutting token consumption 3-10x (up to 23x for WebQA images) while retaining competitive accuracy. On WebQA, image QA F1 rises from 53.0 to 69.4 using only ~83 tokens; full-context baselines collapse at 10,000+ tokens. The post covers training details, interpretability of latent tokens via reconstruction, scaling from 1 to 8 tokens per evidence, and limitations such as lossy compression and the need for paired training data.

One Token per Multimodal Evidence: Latent Memory Rethinks RAG Compression

This zhichai.net post analyzes the paper "One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA" (Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee, arXiv:2606.10572) from the National University of Singapore.

The Problem with Standard RAG

Conventional RAG feeds retrieved evidence verbatim into the context window: a ViT splits one image into 576 visual tokens, and on multimodal WebQA the Full Context baseline exceeds 10,000 tokens—crashing the generator to 0 F1. Retrieval (embedding models like BGE, Qwen-Emb) and generation also operate in two separate representation spaces.

The paper's proposal: compress each evidence item into exactly 1 latent token—a single high-dimensional vector (2048-d text / 4096-d multimodal) used for both retrieval and generation.

How One Token Encodes an Entire Evidence

A small compressor (LLaMA-3.2-1B for text, LLaVA-1.5-7B for multimodal) reads the evidence with a learnable [MEM] token prepended; the last-layer hidden state becomes the latent token. It must simultaneously:

1. Reconstruct the original text/image semantics 2. Retrieve — be comparable to queries in a shared space 3. Answer — drive generation as well as raw evidence

Joint loss: L = 0.5·L_Recon + 0.2·L_Contrast + 1.0·L_Distill

Reconstruction: autoregressive text decoding; for images, reconstruct the CLIP image embedding, then regenerate via a pretrained unCLIP diffusion model.
Contrastive retrieval: latent tokens are projected into a 512-dim shared L2-normalized retrieval space; queries use the same compressor backbone but a separate LoRA adapter.
Distillation (highest weight): a frozen generator (LLaMA-3-8B / Mistral-7B / LLaVA-13B) is trained via KL divergence to behave identically whether prompted with raw evidence or latent tokens (b_z = W_g z with a GELU projection, e.g., 2048→4096). The generator never gets fine-tuned.

At inference, top-k latent tokens are retrieved by inner product and injected directly into inputs_embeds — no raw characters or pixels ever reach the generator, explaining the 3–10x token reduction.

Key Results

Text QA (HotpotQA / 2Wiki / MuSiQue): Latent Memory (k=5) averages ~71 tokens vs ~208–219 for dense baselines; OOD average EM (16.5) trails Qwen3-Emb (24.0) with LLaMA-3-8B but edges it (3.2 vs 2.9) with Mistral-7B — the authors frame this as an efficiency-effectiveness Pareto frontier rather than SOTA.
Multimodal WebQA (the highlight): image QA F1 jumps 53.0 → 69.4 (+31%) while Full Context collapses at 10,013 tokens; average token usage drops to 83 (23x fewer).
Scaling: using 8 latent tokens per evidence (instead of 1) yields linear gains (e.g., 16.5 → 19.9 EM at k=5) for only 35 extra tokens.

Interpretability and Engineering Details

Latent tokens are shown to be semantic carriers, not opaque IDs: text reconstruction (cross-entropy ~0.6–0.7) preserves details like counting three display windows in a Burlington Arcade passage that enabled correct answers; image reconstruction via CLIP embedding + unCLIP preserves semantic content (storefronts, signage, layouts).

Only the compressor LoRAs (r=64, α=128), retrieval MLP, projection W_g, and decoder LoRA are trained (~20h text, ~30h multimodal). Text and image share one latent space, so cross-modal retrieval needs no alignment tricks.

Limitations

Compression is lossy — unsuitable for scenarios needing verbatim citation (legal, medical)
Training requires (question, evidence, answer) triples
Reconstruction is semantic-level, not pixel-perfect
The 512-dim retrieval space is untested at million-document scale (experiments use thousand-level corpora)

Why It Matters

Rather than chasing SOTA, the paper redefines RAG's compression boundary, enabling on-device assistants, real-time multimodal QA over video streams, and compact long-term memory stores. The full pipeline closes in latent space — a strong answer to whether RAG can work under resource constraints.

Paper: arXiv:2606.10572 — "One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA."

Tags: rag, latent-memory, multimodal, llm, on-device-ai, retrieval-augmented-generation, paper-reading, token-compression