Paper Deep Dive: Formalizing Latent Thoughts — An Axiomatic 'Health Check' of LLM Latent Thought Representations

> Paper: Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs > Authors: Fahd Seddik, Fatemeh Fard (University of British Columbia) > Link: https://arxiv.org/abs/2606.27378 > Analysis date: 2026-06-30

TL;DR

This paper gives latent thought representations in LLMs a comprehensive health check, concluding that all mainstream methods fail—they can tell "is this a math or medical question," but cannot distinguish "is 13 prime" from "is 14 prime." More ironically, the embedding of the input prompt itself is more useful than these carefully designed latent representations.

Background

LLM reasoning is shifting from explicit Chain-of-Thought (CoT) toward Latent Thought Representation.

• CoT's problems: generating tokens is slow, verbose, and can leak sensitive information (e.g., o1 being prompted to say "im a grimoire")
• Latent approach: compress reasoning into continuous vectors that "think" directly in vector space without emitting text

Existing methods—Soft Thinking, Latent Thinking, COCONUT—report downstream accuracy gains, but a question arises:

> Does downstream accuracy really mean the representation itself is high quality?

This paper's answer: No. Accuracy can mask severe representational flaws.

Core Contribution: Four Axioms + A Diagnostic Toolkit

The authors propose four functional properties that thought representations must satisfy, independent of representation form (vectors, tensors, sets):

| Axiom | Meaning | Core requirement | |------|---------|-----------------| | Causality | The representation must "substitute" for the reasoning prefix without affecting subsequent generation | Replacing the prefix with T should leave the answer distribution roughly unchanged | | Minimality | Keep only output-relevant information, filter noise | Information bottleneck: minimize I(X;T) while maximizing I(T;Y) | | Separability | Semantically different questions must be distinguishable | A linear classifier can separate output distributions across tasks | | Stability | Robust to lexical variation; reflects model uncertainty | Paraphrases don't change the representation; multimodal outputs retain entropy information |

The authors prove the axioms' logical consistency, independence, and completeness.

Corresponding Quantitative Metrics

| Axiom | Metric | Intuition | |------|--------|-----------| | Causality | KL substitution error: DKL(P(ysuf\|ypre) ‖ P(ysuf\|T)) | How much does the answer distribution change when T replaces ypre? Lower is better | | Minimality | ∆IB = CE(X\|Y,T) − CE(Y\|T) | Does T keep only information useful for predicting Y? Higher is better | | Separability | Same-task / cross-task probe accuracy | Can T distinguish different problem instances? Higher is better | | Stability | DCS (Distributional Consistency Score): AUROC | Can T predict the semantic entropy of the output distribution? Higher is better |

Experimental Findings: Systematic "Representational Collapse"

The authors evaluated 23 tasks from BBEH (Big Bench Extra Hard) across five open-source LLMs:

• Llama-3.1-8B-Instruct
• Llama-3.3-70B-Instruct
• DeepSeek-R1-Distill-Qwen-32B
• Skywork-OR1-32B
• GPT-OSS-20B (MoE)

Candidate representations tested:

• Last Input Token (LIT) at all layers / final layer
• Soft Thinking (ST) at 1/16/32/64/128 steps
• Soft Thinking with Gumbel Noise (STN)
• Latent Thinking (LT) at 1/16/32/64/128 steps
• Output Embedding (OE) — theoretical upper bound
• Input Embedding (IE) — baseline reference

Finding 1: Cross-task separation is fine; within-task separation collapses completely

| Metric | Result | |------|--------| | Cross-task accuracy | Nearly all candidates saturate (math vs. medical questions) | | Same-task accuracy | Nearly all candidates are near random (cannot separate "is 13 prime" from "is 14 prime") |

This is the paper's "fine-grained representational collapse."

> Imagine a medical student who can tell internal medicine from surgery cases, but looking at two different internal medicine patients says "they look the same." That's the current state of latent representations.

Finding 2: The input prompt's embedding outperforms latent representations

The most ironic finding:

> On all four axiom dimensions, no candidate representation consistently beats the Input Embedding.

In other words, the "thought representations" we've labored to design are worse than simply embedding the question text. This suggests an uncomfortable possibility: these latent representations don't actually encode the reasoning process—they may just repeat information already in the input.

Finding 3: More iteration steps can be harmful

For Soft Thinking and Latent Thinking, increasing thinking steps (1 to 128) yields no systematic improvement:

• Stability (DCS): DCS actually *decreases* as steps increase—more iterations encode output-distribution uncertainty worse.
• Separability: within-task discrimination stays near random regardless of step count.

This contradicts intuition: "thinking longer" may just be doing meaningless computation.

Finding 4: A structural flaw, not a model problem

The failure pattern appears consistently across dense, reasoning-distilled, RL-trained, and MoE architectures.

> This means the problem isn't "the model isn't big enough" or "training isn't good enough"—the current design of latent representations itself has a structural flaw.

Deeper Implication: What Are We Actually Optimizing?

The paper's sharpest criticism: existing evaluation conflates "representation quality" with "model capability."

When downstream accuracy is high, we say "this latent representation is good." But the truth may be:

• The model is strong enough that even given garbage representations, the decoder brute-solves the task
• The representation itself may not encode useful reasoning information

It's like judging a student's intelligence by exam scores rather than note quality—but if exams can be cheated (a strong decoder), note quality is hidden. This framework decouples "note quality" from "exam ability" by measuring the representation's functional properties directly, independent of downstream performance.

Audit Results of Existing Methods

| Method | Causality | Minimality | Separability | Stability | Overall | |--------|-----------|------------|--------------|-----------|---------| | Soft Thinking (ST) | ≈ IE | slightly > IE | cross-task ✓, same-task ✗ | declines with steps | underwhelming | | Soft Thinking + Noise (STN) | ≈ IE | slightly > IE | cross-task ✓, same-task ✗ | worst | noise hurts stability | | Latent Thinking (LT) | ≈ IE | ≈ IE | cross-task ✓, same-task ✗ | declines with steps | nearly indistinguishable from IE | | Last Input Token (LIT) | ≈ IE | < IE | cross-task ✓, same-task ✗ | ≈ IE | worse than just using IE | | Input Embedding (IE) | baseline | baseline | cross-task ✓, same-task ✗ | baseline | surprisingly strong | | Output Embedding (OE) | theoretical upper bound | N/A | only same-task > random | N/A | naturally strong since it knows the answer |

Limitations and Future Directions

The authors candidly list limitations:

1. The lexical invariance sub-property of Stability is untested: all candidates are trivially invariant to paraphrase (identical vectors). 2. Computational cost exceeds a single benchmark run: requires output generation and probe training, though the information value justifies it. 3. Covers only 23 BBEH reasoning tasks: multilingual and non-reasoning scenarios unexplored. 4. All candidate representations come from pretrained models: representations specifically trained to satisfy the four axioms could be future work.

My Take

This paper's academic value is high because it:

1. Establishes the first principled evaluation framework: axiomatic definitions + quantitative metrics instead of heuristic proxies 2. Reveals the truth masked by accuracy: high downstream scores ≠ good representation quality 3. Gives a clear optimization target: the four axioms can serve as training objectives, not just evaluation metrics

But practical impact may be limited—it's more of a "health report" telling you "the patient has these problems" without prescribing a cure. Future work must tackle "how to design representations satisfying the four axioms."

An interesting extension: if input prompt embeddings are already this strong, what exactly is the value of latent reasoning?

Possible answers:

• Compression: for long CoT, latent representations can dramatically reduce token counts
• Privacy: no exposed reasoning traces
• Multiple paths: one latent vector can encode a superposition of reasoning paths

But if these representations genuinely cannot distinguish different instances of the same task, their "reasoning" component is highly suspect.

Key Quotes

> "No candidate beats the Input Embedding reference on every axis across the tested LLMs."

> "The representations encode little information beyond what is already present in the input embedding."

> "The failure is consistent across dense, reasoning-distilled, and RL-trained model families, indicating that the gap is structural rather than a property of model size or training procedure."