ToM2Vec: Continuous Latent Tokens Enable Theory of Mind for Multi-Agent LLM Coordination

Forum topic · ✨步子哥 · 2026-10-22

Summary

ToM2Vec is a new approach that gives LLM agents a form of 'mind reading' in multi-agent settings by predicting teammates' mental states in a continuous latent space instead of natural language. The key insight: text is discrete and commits to a single hypothesis, while continuous latent tokens can hold multiple hypotheses simultaneously. Trained via self-play with Qwen3-8B, an agent learns to emit K=10 latent tokens aligned (via cosine similarity loss) with hidden states extracted from a partner's chain-of-thought reasoning. At inference, these tokens condition the agent's own decisions without any access to the partner's private information. ToM2Vec improves results across Overcooked, Hanabi, Decrypto, negotiation, and collaborative escape/capture tasks, outperforming language-based Theory of Mind baselines—some of which score worse than no ToM at all. Notably, the skill transfers across models: training only with Qwen3-8B still yields +17.3 average points when paired with GPT-5.4. Probe experiments (belief/desire/action questions, context-corruption perplexity tests) confirm the tokens encode partner-specific reasoning rather than generic coordination signals. Code is open-sourced at github.com/nsivaku/tom2vec.

You Can Never Guess What Your Teammate Is Thinking — Unless You Can Read Minds: ToM2Vec

An Awkward Moment in the Kitchen

Imagine you're in an Overcooked kitchen. You're chopping onions; your teammate is making soup. You place the chopped onions on the counter and wait — but your teammate doesn't move.

What are you thinking? "Did they not see the onions? Are they waiting for me to deliver them? Or are they going to grab plates first?"

Three possibilities surface at once. You don't pick one before acting — you observe while holding all three, waiting for the next signal to narrow things down.

Humans do this naturally. We don't lock a teammate into a single state the moment we see one action. We maintain multiple hypotheses and let evidence eliminate the wrong ones.

LLM agents can't do this.

Text: An Overly Committed Medium

The standard approach for making LLM agents reason about teammate intent is prompting: "Describe what your teammate might be thinking."

The problem: text is discrete, deterministic, and can only say one thing at a time. When an LLM writes "the teammate probably wants to grab the onions," it has collapsed a probability distribution into a single sentence. The other possibilities — waiting for plates, waiting on routing, not noticing at all — are squeezed out.

This isn't bad prompting. It's a limitation of text as a representation.

Worse, researchers found that making an LLM verbalize a teammate's reasoning (Language ToM training) is sometimes worse than doing no ToM at all. On the Coordination Ring map, Language ToM dropped scores by 0.7 points; on Collab Escape it fell 10 percentage points.

Language is not the right vehicle for mind reading.

ToM2Vec: Holding Multiple Hypotheses in Latent Space

ToM2Vec's core insight comes from an analogy: latent variable reasoning. Like Coconut (chain of continuous thought) and other latent reasoning work, continuous latent variables can "superimpose" multiple reasoning paths instead of committing to one as text CoT does. ToM2Vec applies this to reading teammates.

Training: Teaching the Agent to "Read" the Partner

1. The partner runs its reasoning: during training, a partner agent (the same Qwen3-8B) generates its chain-of-thought trace normally. 2. Extract the partner's "brainwaves": the trace is split at sentence boundaries; hidden states (last-layer vectors) are extracted at each boundary. These vectors are the latent-space representation of what the partner is thinking. 3. Train the agent to predict these vectors: the agent sees only its own local observations (not the partner's private information) and generates K=10 continuous latent tokens, trained with a cosine-similarity loss against the partner's hidden states.

In other words: the agent learns to predict its teammate's "brainwaves" from its own perspective — not what the partner will say, but what the vectors inside the partner's mind look like.

Inference: The Agent Reads Minds on Its Own

At test time, the partner's private information is entirely invisible. The agent generates 10 latent tokens from its own observations and interaction history — its "guess" of the partner's mental state — and uses them as a prefix to condition its subsequent reasoning and actions.

No partner needed. No partner reasoning trace. Just the agent's own observations.

Why Exactly 10 Tokens?

K=10 is deliberate: 2 tokens mark the start and end of reasoning; 8 cover the middle segments. The partner's trace is split at sentence boundaries and 8 positions are sampled uniformly for hidden-state extraction, ensuring the agent learns a compressed representation of the entire reasoning process — not just its beginning or end.

Five Arenas: From Cooperation to Competition

| Environment | Type | Baseline | ToM2Vec | Gain | |---|---|---|---|---| | Overcooked (Cramped Room) | Cooperative | 89 | 98 | +9 | | Overcooked (Asymmetric) | Cooperative | 93 | 108 | +15 | | Overcooked (Coordination Ring) | Cooperative | 105 | 113 | +8 | | Negotiation | Mixed-motive | 62% deal rate | 71% deal rate | +9pp | | Decrypto (encryptor) | Cooperative + competitive | — | +4.4pp win rate | — | | Decrypto (interceptor) | Competitive | — | +6.1pp win rate | — | | Hanabi | Cooperative (hidden info) | 8.9 | 11.3 | +2.4 | | Collab Capture | Cooperative | 0.67 capture rate | 1.00 capture rate | +33pp | | Collab Escape | Cooperative | 0.76 escape rate | 0.92 escape rate | +16pp |

A clean sweep. But the most convincing evidence comes from the following findings.

Finding 1: Language-Based Mind Reading Underperforms Latent Mind Reading

In a key ablation, ToM2Vec's latent tokens were replaced with natural-language summaries (Language ToM training) — the agent describes the teammate's thoughts in words, then conditions on that text.

Result: Language ToM underperformed ToM2Vec in every environment; in some it was worse than no ToM at all.

Why? Language is discrete and expresses one hypothesis at a time. Writing "the teammate wants the onions" collapses the distribution. If the teammate was actually waiting for plates, this wrong commitment leads to bad decisions.

Latent tokens are continuous and can encode a superposition of hypotheses. The agent doesn't need to "pick one before acting" — it can keep observing and deciding under uncertainty.

This echoes Coconut: language is not the only medium of thought. Some reasoning is more natural in latent space.

Finding 2: Mind Reading Transfers Across Models

ToM2Vec was trained only with Qwen3-8B as the partner (self-play). At test time, two unseen partners were swapped in:

| Partner | Baseline avg | ToM2Vec avg | Gain | |---|---|---|---| | GPT-5.4 | 135/193/174 | 152/214/188 | +17.3 | | Gemma-4 | 65/90/87 | 68/105/94 | +3~15 |

Paired with GPT-5.4, ToM2Vec gained +17.3 points on average across three maps — despite GPT-5.4 being far stronger than the training partner.

This shows ToM2Vec learned not "Qwen3-8B's specific reasoning patterns" but "the general structure of agent reasoning" — a cross-model partner model. The engineering implication is huge: train ToM ability with an open-source model, then deploy alongside closed-source models.

Finding 3: It's Actually Reading Minds, Not Learning Shortcuts

Coordination gains alone don't prove the tokens encode Theory of Mind. Three probe experiments verify this:

Bayesian ToM Probe

Across 300 Overcooked states, the agent answers questions about the partner's:

Beliefs: what does the partner think I'm doing? (second-order — hardest)
Desires: what does the partner want now?
Actions: what will the partner do next?

| Method | Belief | Desire | Action | |---|---|---|---| | Random | 16.7% | 16.7% | 16.7% | | Untrained | 41.7% | 53.0% | 55.3% | | Prompted ToM | 40.0% | 55.0% | 56.3% | | Language ToM | 46.7% | 59.3% | 61.3% | | SimToM | 43.7% | 54.7% | 58.0% | | Adaptive ToM | 45.7% | 60.3% | 60.0% | | ToM2Vec | 52.3% | 62.3% | 64.0% |

ToM2Vec is best on all three dimensions, leading by 5.6pp on the hardest (second-order beliefs).

Context Corruption

If the tokens encode the partner's reasoning, corrupting the context should degrade predictions of that reasoning. Two corruptions were tested:

Swapped layout: replacing the kitchen layout with another map's
Adversarial intent: assuming the partner wants to sabotage

Latent tokens from normal contexts achieved perplexity 1.412 on the partner's true reasoning, vs 1.702 and 1.613 for corrupted contexts. Normal contexts gave the lowest perplexity 98.6% of the time.

The tokens encode partner-specific reasoning, not generic coordination signals.

Engineering Insights

1. Training vs inference cost: ToM2Vec requires training (unlike plug-and-play prompting), but only via self-play with no human labels. Inference adds just 10 latent tokens of forward computation — nearly free. 2. Self-play training, cross-model deployment: trained with Qwen3-8B self-play, still effective with GPT-5.4. Any team can train ToM with open models and deploy with closed ones. 3. K=10 as a sweet spot: enough to cover multiple stages of the partner's reasoning (start, middle segments, end) without destabilizing training. In code: DEFAULT_K_LATENTS = 10, with 2 for start/end and 8 for the middle. 4. Cosine similarity over MSE: the objective is directional alignment, not exact replication of hidden states — fitting for semantic representations where direction matters more than magnitude.

Open-Source Code

Full code: https://github.com/nsivaku/tom2vec

tom2vec/model.py: the model class, centered on emit_latents (generating K latent tokens) and extract_targets (extracting training targets from partner reasoning)
tom2vec/loss.py: cosine-similarity loss, ~5 lines of core code
tom2vec/chunking.py: sentence-boundary splitting and position sampling
envs/: implementations of the five environments (Overcooked, Hanabi, Decrypto, negotiation, Collab Escape/Capture)

Reflections

Another Case of "Scalar Illusion"

ToM2Vec's finding condenses to: textual ToM is a scalar; latent ToM is a distribution. Describing a teammate's thoughts in words collapses a probability distribution into a point. Textual ToM assumes a mental state can be summarized in one sentence, when in reality it's a distribution that needs a vector space. This is the same class of problem as accuracy hiding understanding vs. decision errors: scalar evaluation hides high-dimensional structure.

Judgment–Gate Decoupling

The architecture decouples judgment (latent token generation — inferring the partner's mental state) from the gate (action selection conditioned on those tokens). Trained and executed separately, "mind reading" and "acting" can each be optimized without interference.

Mind Reading as a Training Objective

Most exciting is the training paradigm itself: making "reading the partner's mind" an explicit training objective. Previous ToM research either tested whether LLMs have ToM (Sally-Anne tests) or prompted them to consider teammate intent. ToM2Vec makes ToM a trainable skill, generalizable to any task requiring modeling another agent — teammate, opponent, or user.

Language Is Not the Only Medium of Thought

Together with Coconut and HybridAgent, ToM2Vec points one direction: some reasoning — especially reasoning that must hold multiple hypotheses at once — is more natural in latent space. Theory of Mind is the paradigm case: you hold "the partner might be thinking A / B / C" and let evidence converge. Perhaps human ToM is powerful precisely because it operates beneath language — we simultaneously "feel" multiple possible states before verbalizing the most likely one. ToM2Vec partially reconstructs that sub-linguistic layer.

---

Paper: ToM2Vec: Continuous Theory-of-Mind Tokens for Multi-Agent Coordination (arXiv: 2610.29934) Code: https://github.com/nsivaku/tom2vec Model: Qwen3-8B, self-play training, 10 latent tokens