LatComm: One Model Reads the Document, Another Model Reads Its "Memory"

*Translated and adapted from a zhichai.net forum post.*

An Overlooked Cost in Multi-LLM Systems

Consider a system where multiple LLMs collaborate—one for retrieval, one for reasoning, one for generation—sharing the same long context: a 50-page contract, a technical manual, or hours of meeting transcripts.

The current approach passes that text as plain text to every model. Every new question forces the receiving model to reprocess all tokens of the entire context. With a 100k-token context used by three models, each question burns 300k tokens of compute. Worse, the text is identical for every question, yet each model re-reads it from scratch.

It's like three translators taking turns on the same book, each starting over every time someone else picks it up. Can the first model pass its "understanding" directly to the next?

Why Existing Approaches Fall Short

Context compression methods (e.g., ICAE, AutoCompressor) compress long text into few tokens, but the compressed representation can only be read by the same model—different models have different embedding spaces.

Latent communication methods (e.g., cross-attention based latent communication) let one model pass internal states to another, but typically require joint training of both models, and re-encode for every question since encoding depends on the query.

In other words, compact, reusable, and cross-model readable—existing methods satisfy at most two of these three properties simultaneously.

LatComm's Solution: Decoupling Encoding from Reading

LatComm (Latent Communication interface) completely separates context encoding from context reading:

1. Encoding phase: A frozen context encoder compresses the long context into a fixed-size "packet." Encoding depends only on the context itself, not on any question. 2. Reading phase: A frozen receiver LLM reads from the packet via cross-attention. Crucially, the receiver computes "gates" from its own current prompt, deciding which parts of the packet to read.

The elegance: the packet is encoded once but read repeatedly for different questions. Since both encoder and receiver are frozen, no joint training is needed—any two LLMs can use it directly.

An Analogy: Library and Readers

• Traditional: every reader re-reads the whole book for each question—slow.
• Context compression: a librarian summarizes the book, but only readers trained by that same librarian understand the summary; a new librarian must re-summarize.
• Latent communication: librarian and readers are co-trained on the "secret codes," but every new question forces the librarian to re-encode.
• LatComm: the librarian encodes the book into a one-time "structured index" (packet). Each reader, guided by their own question, uses a retriever (gates) to pick relevant parts of the index. Different readers read different parts; the index never changes.

Results: 57x Compression, 3x Speedup

Across 12 multiple-choice QA datasets, LatComm improves average accuracy by up to 10.2 percentage points over single-vector baselines.

On QuALITY—a long-text reading comprehension benchmark with articles averaging ~5,000 words—LatComm reduces the receiver's input by 57x and speeds up answering all questions per article by 3x.

To put 57x in perspective: 5,000 tokens of input becomes a packet of roughly 88 tokens. And that packet is reused across all questions on the same article—encoded once, read repeatedly.

Why It Matters

Multi-model systems are increasingly common, from RAG to agent workflows, and repeatedly reprocessing full text remains a pain point. LatComm shows that context doesn't need to be re-read—it can be encoded into a reusable "memory format" that different models read on demand.

It extends the "retrieve instead of read" philosophy: rather than giving models more information, give them the right information—but LatComm moves that "right information" from external retrieval to a shared inter-model memory layer.

Limitations and Reflections

• The encoder and receiver must come from the same architecture family (e.g., both LLaMA-series); cross-architecture compatibility is unverified.
• Fixed-size packets mean lossy compression—tasks requiring verbatim quoting (e.g., legal document review) may lose critical details.
• The gating mechanism adds per-question compute to the receiver; total throughput is still faster than full-text processing, but marginal per-question cost is slightly higher than plain text input.

As a proof of concept, though, LatComm demonstrates that different LLMs can share compressed memory without joint training—opening a new direction for efficiency in multi-model systems.

Next time you see three LLMs taking turns processing the same long text, ask: are they each "reading" the document, or sharing one "memory"?

---

Paper: LatComm: Reusable Context Compression for Latent Communication between Large Language Models Authors: Huawei Lin, Jinting Liu, Tan Yu arXiv: https://arxiv.org/abs/2610.26840 HTML: https://arxiv.org/html/2610.26840v1