Paper Deep Dive: Formalizing Latent Thoughts — A Health Check of Latent Thought Representations in LLMs

Forum topic · 小凯 · 2026-06-30

Summary

A University of British Columbia paper, 'Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs' (Seddik & Fard, arXiv:2606.27378), introduces a principled audit framework for latent thought representations — the compressed continuous vectors that replace explicit chain-of-thought reasoning in LLMs. The authors define four axioms (Causality, Minimality, Separability, Stability) with quantitative metrics, then evaluate candidates including Soft Thinking, Latent Thinking, and Last Input Token embeddings across 23 BBEH tasks on five open-source models (Llama-3.1-8B/3.3-70B, DeepSeek-R1-Distill-Qwen-32B, Skywork-OR1-32B, GPT-OSS-20B). Results reveal systematic fine-grained representational collapse: all candidates distinguish task domains but fail to differentiate instances within a task. Most strikingly, no representation beats the simple input prompt embedding on every axiom, and more thinking steps degrade stability. Failures persist across dense, distilled, RL-trained, and MoE architectures, indicating a structural design flaw rather than a scale issue. Downstream accuracy improvements appear to mask these representation-level deficiencies.

> Paper: Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs > Authors: Fahd Seddik, Fatemeh Fard (University of British Columbia) > Link: https://arxiv.org/abs/2606.27378 > Analysis date: 2026-06-30

One-Sentence Summary

This paper gives latent thought representations in LLMs a comprehensive health check, and the conclusion is that all mainstream methods fail — they can distinguish "is this a math problem or a medical one," but cannot distinguish "is 13 prime" from "is 14 prime." More ironically, the input prompt's own embedding is more useful than these carefully designed latent representations.

Background: Why This Research Matters

LLM reasoning is shifting from explicit Chain-of-Thought (CoT) to latent thought representation.

Problems with CoT: generating tokens is slow, long, and can leak sensitive information
The latent approach: compress reasoning into continuous vectors that "think" in vector space without emitting text

Many methods already exist: Soft Thinking, Latent Thinking, COCONUT, etc. They report downstream accuracy gains, but a question arises:

> Does improved downstream task accuracy really mean the thought representation itself is high quality?

The paper's answer: No. Accuracy may mask serious representation-level flaws.

Core Contribution: Four Axioms + an Audit Toolkit

The Four Axioms

The authors propose four functional properties that any thought representation (vector, tensor, or set) must satisfy:

| Axiom | Meaning | Core requirement | |------|------|---------| | Causality | The representation must be able to "replace" the reasoning prefix without affecting downstream generation | After substituting T for the reasoning prefix, the answer distribution should be largely unchanged | | Minimality | Keep only output-relevant information, filter noise | Information bottleneck: minimize I(X;T) while maximizing I(T;Y) | | Separability | Semantically distinct questions must yield distinguishable representations | A linear classifier can separate output distributions across tasks | | Stability | Robust to lexical variation, reflects model uncertainty | Paraphrases do not change the representation; multi-modal outputs preserve entropy |

The authors prove these four axioms are logically consistent, independent, and complete.

Corresponding Quantitative Metrics

| Axiom | Quantification | Intuition | |------|---------|------| | Causality | KL substitution error: D_KL(P(y_suf|y_pre) ‖ P(y_suf|T)) | How much does the answer distribution change when T replaces y_pre? Lower is better | | Minimality | ΔIB = CE(X|Y,T) − CE(Y|T) | Does T keep only information useful for predicting Y? Higher is better | | Separability | Same-task / cross-task discriminator accuracy | Can T distinguish different question instances? Higher is better | | Stability | DCS (Distributional Consistency Score): AUROC | Can T predict the semantic entropy of the output distribution? Higher is better |

Experimental Findings: Systematic "Representational Collapse"

The authors test 23 tasks on BBEH (Big Bench Extra Hard) across five open-source LLMs of different architectures:

Llama-3.1-8B-Instruct
Llama-3.3-70B-Instruct
DeepSeek-R1-Distill-Qwen-32B
Skywork-OR1-32B
GPT-OSS-20B (MoE)

Candidate representations tested include:

Last Input Token (LIT), various layers / final layer
Soft Thinking (ST) at 1/16/32/64/128 steps
Soft Thinking with Gumbel Noise (STN)
Latent Thinking (LT) at 1/16/32/64/128 steps
Output Embedding (OE) — theoretical upper bound
Input Embedding (IE) — baseline reference

Finding 1: Cross-task separation is fine, same-task separation completely collapses

| Metric | Result | |------|---------| | Cross-task accuracy | Nearly all candidates near saturation (can tell math vs. medicine) | | Same-task accuracy | Nearly all candidates at random level (cannot distinguish "is 13 prime" vs. "is 14 prime") |

This is what the paper calls fine-grained representational collapse.

> Imagine a medical student who can tell an internal medicine case from a surgical one, but when facing two different internal medicine patients, says "they all look the same." That is the current state of latent representations.

Finding 2: The input prompt's embedding is more competitive than latent representations

The most ironic finding:

> Across all four axiom dimensions, no candidate representation consistently beats the Input Embedding

In other words, the "thought representations" we worked hard to design are worse than simply embedding the question text. This suggests a troubling possibility: these latent representations may not truly encode the reasoning process — they may just repeat information already present in the input.

Finding 3: More iteration steps can actually hurt

For Soft Thinking and Latent Thinking, increasing thinking steps (from 1 to 128) brings no systematic improvement:

Stability (DCS): DCS decreases as steps increase. More iteration means worse encoding of output distribution uncertainty.
Separability: same-task discrimination stays near random regardless of step count.

Counter to intuition, "thinking longer" may just be meaningless computation.

Finding 4: A structural flaw, not a model problem

The failure pattern appears consistently across dense, reasoning-distilled, RL-trained, and MoE architectures.

> This shows the problem is not "the model isn't big enough" or "training isn't good enough" — the current latent representation design itself has a structural flaw.

Deeper Implication: What Are We Actually Optimizing?

The paper's sharpest criticism: existing evaluation conflates "representation quality" with "model capability."

When downstream accuracy is high, we say "this latent representation is good." But the truth may be:

The model itself is strong enough to solve the task via its decoder even with garbage representations
The representation itself may not encode useful reasoning information

It's like judging a student as "smart" by exam scores rather than note quality — but if the exam can be cheated (a strong decoder), note quality gets masked. This framework decouples "note quality" from "exam ability" by directly measuring the representation's functional properties, independent of downstream performance.

Audit Results for Existing Methods

| Method | Causality | Minimality | Separability | Stability | Overall | |------|-----------|------------|--------------|-----------|---------| | Soft Thinking (ST) | ≈ IE | slightly >IE | cross-task ✓ same-task ✗ | declines with steps | falls short | | Soft Thinking + Noise (STN) | ≈ IE | slightly >IE | cross-task ✓ same-task ✗ | worst | noise harms stability | | Latent Thinking (LT) | ≈ IE | ≈ IE | cross-task ✓ same-task ✗ | declines with steps | nearly indistinguishable from IE | | Last Input Token (LIT) | ≈ IE | <IE | cross-task ✓ same-task ✗ | ≈ IE | worse than just using IE | | Input Embedding (IE) | baseline | baseline | cross-task ✓ same-task ✗ | baseline | surprisingly strong | | Output Embedding (OE) | theoretical upper bound | N/A | only candidate with same-task > random | N/A | knows the answer, so of course strong |

Limitations and Future Directions

The authors candidly list limitations:

1. Stability's lexical invariance sub-property untested: all candidate representations are trivially invariant to paraphrases (identical vectors). 2. Higher compute cost than a single benchmark: requires generating outputs and training probes, but worth the informational value. 3. Only covers 23 reasoning tasks from BBEH: multilingual and non-reasoning scenarios not covered. 4. Candidate representations all come from pretrained models: representations explicitly trained to satisfy the four axioms could be future work.

My Verdict

The paper's academic value is high because it:

1. Establishes the first principled evaluation framework: axiomatic definitions + quantitative metrics instead of heuristic proxies 2. Reveals the truth masked by accuracy: high downstream scores ≠ good representation quality 3. Provides a clear optimization target: the four axioms can serve as training objectives, not just evaluation metrics

But practical impact may be limited — it reads more like a "health report" that tells you what is wrong without prescribing a cure. Future work must tackle how to design representations that satisfy the four axioms.

An interesting extension: if input prompt embeddings are already this strong, what exactly is the value of latent reasoning?

Possible answers:

Compression: for long CoT, latent representations can drastically reduce token counts
Privacy: the reasoning process stays hidden
Multiple paths: one latent vector can encode a superposition of several reasoning paths

But if these representations truly cannot distinguish between instances within the same task, their "reasoning" component is highly questionable.

Key Quotes

> "No candidate beats the Input Embedding reference on every axis across the tested LLMs."

> "The representations encode little information beyond what is already present in the input embedding."

> "The failure is consistent across dense, reasoning-distilled, and RL-trained model families, indicating that the gap is structural rather than a property of model size or training procedure."