Every Token Leaves a Ripple: Finding the Truly Important Tokens in Chain-of-Thought via the Residual Stream

Forum topic · 小凯 · 2026-09-01

Summary

This post analyzes the paper 'Every Token Leaves a Ripple in the Stream of Thought', which introduces MIST, a model-internal method for scoring token saliency in chain-of-thought (CoT) reasoning. Instead of relying on external scorers or attention weights, MIST reads token importance directly from the transformer's residual stream, defining two complementary axes: necessity (how much answer log-likelihood drops when a token's residual contribution is ablated) and sufficiency (how much answer signal is restored when only that token's contribution remains). Experiments on 100 MATH reasoning chains with Qwen2.5-1.5B-Instruct show these axes are nearly uncorrelated (Spearman rho = -0.07, top-30% overlap 0.28), revealing that CoT structure is a network rather than a linear sequence of steps. Both axes are computable with one backward pass each via first-order Taylor linearization, reducing cost from O(T) forward passes to O(1). MIST outperforms baselines including TokenSkip, perplexity, attention rollout, and H2O across GSM8K, MATH, MMLU-Pro, and BBH-MC on four models. The post frames this as evidence that 'importance' inside LLMs is a vector, not a scalar.

Overview

This post discusses the paper *Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression* (arXiv:2608.31066), by Tianyi Zhao, Yinhan He, Wendy Zheng, and Chen Chen (University of Virginia). The paper proposes MIST (Model-Internal Saliency for Token-level CoT compression), a method that identifies which tokens in a chain-of-thought actually matter by reading the model's own internal computations.

The problem

Given a 500-token chain-of-thought for a math problem, suppose you want to compress it to 30% while preserving accuracy. Which 150 tokens should you keep? Existing methods use external scorers (another LLM judging importance) or attention weights as proxies — all of them are outsiders guessing what the model itself considers important. MIST's core idea: ask the model directly.

Key points

Residual stream as the signal source: Transformer layers add information to a residual stream rather than replacing it. The paper's insight is that the size of the "ripple" a token leaves in this stream reflects its contribution to computing the answer. This is a computable quantity, not a metaphor.
Two axes of importance:
*Necessity*: how much does answer log-likelihood drop if a token's residual contribution is erased? ("Can we do without it?")
*Sufficiency*: how much answer log-likelihood is recovered if only that token's residual contribution remains? ("Is it enough on its own?")
The axes are nearly uncorrelated: On 100 MATH reasoning chains with Qwen2.5-1.5B-Instruct, the Spearman correlation between necessity and sufficiency is -0.07, with only 0.28 overlap in the top-30%. High-necessity tokens are "context-dependent" load-bearing connectors; high-sufficiency tokens are "self-contained" key insights. Using either axis alone fails at different compression budgets; MIST ranks tokens by a unified necessity × sufficiency score.
Engineering breakthrough: Direct computation would require one intervention per token (O(T) forward passes). The paper shows both axes admit first-order Taylor linearization: each becomes an inner product between the token's residual contribution vector and the gradient of answer log-likelihood w.r.t. the layer's residual. Two backward passes total — O(1) regardless of chain length. Appendix D proves the remainder is bounded under regularity assumptions, and D.5 empirically verifies rank consistency with exact interventions.
Results: Across 4 benchmarks (GSM8K, MATH, MMLU-Pro, BBH-MC) and 4 models, MIST beats all baselines (TokenSkip, perplexity, attention rollout, H2O, uniform sampling, no-chain) at every compression budget, with significant gains over TokenSkip at 30% retention on MATH. Ablations confirm both axes are indispensable.

Interpretation

The author connects this to prior findings (SCIT on latent reasoning; TwinKV showing attention weights and causal contribution are nearly uncorrelated, rho = -0.004). Together these suggest "importance" inside LLMs is a vector, not a scalar — any method ranking tokens along a single dimension will fail in some regimes (a "scalar illusion": measuring blood pressure with a thermometer).

Deeper implication: CoT is not a line but a network. Compression is not "delete redundant steps" but "balance network density and connectivity." The necessity-sufficiency separation itself stems from Transformer nonlinearity — in a linear system the two measurements would be equivalent (superposition principle); their divergence quantifies how deeply a token participates in nonlinear computation.

Limitations and open questions

1. Requires gradient and residual-stream access — not applicable to closed-source API models. 2. Validated mainly on math/reasoning benchmarks; code and long-form domains untested. 3. Taylor linearization is an approximation and may be inaccurate for extreme residual contributions.

The key open question: is the necessity/sufficiency separation an intrinsic structure of CoT, or an artifact of current training paradigms? Would it disappear if future models learned more compact reasoning?

---

Paper: Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression

Authors: Tianyi Zhao, Yinhan He, Wendy Zheng, Chen Chen (University of Virginia)

Method: MIST (Model-Internal Saliency for Token-level CoT compression)

Key data: Necessity-sufficiency Spearman rho = -0.07, top-30% overlap 0.28; consistently outperforms baselines across 4 benchmarks and 4 models.