*Translation of a forum post discussing the paper "Tools Need Not Speak in Text: Latent Context Compression for Agents" (arXiv 2611.06500) by the THU-Agent team at Tsinghua University.*
It opens with an analogy: a translator stands between you and a friend who actually speaks your language. You never realized direct communication was possible — until someone points out that every translation, fee, and loss of nuance was unnecessary. In today's AI agents, that "translator" is text.
Why must tools speak human language?
Consider asking an AI for Beijing's weather. It calls an API that returns a JSON string like {"city": "Beijing", "date": "2026-11-07", "high": 15, "low": 3, "condition": "sunny", ...} — roughly 120 tokens. The AI must then read this JSON *as text*, attend to every quote, colon, and field name, consume context window space, inflate the KV cache, and slow down the whole system.
In real agent settings — web browsing on WebArena, multi-step reasoning on GAIA — a single task may involve dozens of tool calls, each returning thousands of tokens: full HTML pages, complete execution logs. These observations pile up and cause "context rot": attention budgets get consumed by low-value tokens, models lose track of earlier context, and error rates rise.
Existing solutions all compress *text* — summarization, key-field extraction, delimiters. But the fundamental limit is that text itself is an inefficient carrier for machines, designed for human reading and writing, not for information efficiency. A 120-token JSON may contain only a dozen useful concepts.
Latent space: AI's native language
Transformers don't process text as discrete symbols internally; they map tokens into a continuous, high-dimensional latent space, where semantic similarity becomes geometric proximity. If text is spoken language, latent space is telepathy — no encoding/decoding loss.
The paper, titled Tools Need Not Speak in Text, makes tool observations enter the model directly as latent vectors in the KV cache — the cached key/value vectors that constitute the model's working memory. Since the KV cache is already made of vectors, why route observations through the detour of text → tokenization → embeddings? Coca-Cola skips it.
Two key components:
- Latent Invention: a lightweight encoder compresses raw observations into latent tokens — continuous vectors carrying full semantics at a fraction of the token count — injected directly as KV cache entries, like implanting artificial memories.
- Action-Preserving Training: training ensures decision-critical signals (information that changes the agent's next action) survive compression, while formatting-level redundancy can be aggressively compressed or discarded — like a translator who guarantees key contract clauses but may drop boilerplate.
- 70.01%–89.24% token reduction
- 24.45%–55.97% wall-clock time reduction
- Performance maintained or improved on GAIA and WebArena; some GAIA subsets even beat the uncompressed baseline
- Tools Need Not Speak in Text: Latent Context Compression for Agents. arXiv:2611.06500, THU-Agent Team, Jia Li et al., Tsinghua University.
- Project page and code: github.com/THU-Agents/coca-cola
From the model's perspective, latent tokens are indistinguishable from ordinary KV entries, so no architecture changes are needed — only input-side processing. Coca-Cola applies to any LLM with a suitable encoder.
Results: fewer tokens, better performance
Counterintuitively, compression doesn't hurt performance:
Possible explanations:
1. Attention purification — latent tokens are pre-digested information; the model no longer sifts noise and can focus on decisions. 2. Context window decompression — effective capacity multiplies; long multi-step tasks no longer push early context out of attention. 3. Implicit regularization — latent representations filter surface-level details (exact numbers, phrasing, formatting) and preserve semantic abstractions, yielding more robust decisions.
Limitations: the encoder must be trained per domain (cross-domain generalization is unproven), and fine-grained precision (e.g., exact numerical computations) can be lost — Action-Preserving Training mitigates but does not eliminate this.
The bigger narrative
Placed alongside two companion papers, a story emerges across three layers of AI infrastructure: model level (CompressionBench — smaller models), system level (ServerBench — faster serving, e.g., 4.87x vLLM throughput on Gemma E4B for Claude Sonnet 5.5), and now the interface level (Coca-Cola — efficient communication). Together: AI is gaining full-stack ability to reshape its own runtime environment — recursive self-improvement moving from sci-fi to engineering reality.
Deeper implications
Coca-Cola opens the door to non-textual agent communication protocols. Tool developers might stop optimizing for human-readable JSON and instead design encoder-friendly information structures. Agent-to-agent communication could eventually bypass natural language entirely.
But there is a price: latent representations are far less interpretable. You cannot read a latent token to audit what the agent "saw," making debugging, transparency, and accountability genuinely harder. It is a transfer of power — from human-auditable text to vectors only the model can verify.
AI's native language is neither English nor Chinese — it is a continuous, high-dimensional, mathematical space. Teaching tools to speak it doesn't just optimize agents; it lets AI think in its own way.
The translator can rest now.
---
References