Every Token Leaves a Ripple: Finding Truly Important Tokens in CoT via the Residual Stream

A Concrete Scenario

You have a reasoning LLM. You give it a math problem and it produces a 500-token Chain-of-Thought (CoT), then an answer. You want to compress the chain—keep only 30% of the tokens while the model still answers correctly.

The question is: which 150 tokens should you keep?

Existing methods score tokens with external scorers—another LLM judging each token's importance, or attention weights as a proxy. These work, but they all guess at "what the model itself considers important" from an outsider's perspective.

The paper *Every Token Leaves a Ripple in the Stream of Thought* asks a more direct question: why not just ask the model itself?

The Residual Stream: The Model's "River of Thought"

To understand the method, first consider a key internal structure—the residual stream.

Each Transformer layer doesn't replace information; it adds to it. Token embeddings enter the residual stream, each layer's output is added into it, and the stream's final content is used to predict the next token. Think of the residual stream as a river: each token's embedding is the source, each layer pours water in, and the final river state determines the output.

The paper's core insight: the size of the "ripple" each reasoning token leaves in this river reflects its contribution to computing the answer.

• A token that causes large perturbations in the residual stream, repeatedly used by later layers, is critical to the answer.
• A token whose ripple is faint and barely referenced downstream is redundant and can be safely removed.

This is not a metaphor—it is a computable quantity.

Two Axes: Necessity and Sufficiency

The paper defines token importance along two complementary axes:

Necessity: If a token's residual contribution is erased, how much does the answer's log-likelihood drop? This measures "can we do without it?"

Sufficiency: If only this token's residual contribution is provided (all other reasoning tokens erased), how much of the answer's log-likelihood is recovered? This measures "is it alone enough?"

The two axes capture different things:

• High necessity = the token is a load-bearing wall; remove it and the chain collapses
• High sufficiency = the token is a key hub; providing it alone restores substantial answer signal

Key finding: the two axes are almost uncorrelated. On 100 MATH reasoning chains with Qwen2.5-1.5B-Instruct, the Spearman correlation between necessity and sufficiency is only -0.07, and the top-30% overlap is only 0.28.

This means: the most "necessary" tokens and the most "sufficient" tokens are two different sets of tokens.

Why the Axes Separate

The separation seems counterintuitive—if a token matters to the answer, shouldn't it be both necessary and sufficient? But it reveals the deep structure of CoT:

High-necessity tokens are context-dependent. They matter because their combination with other tokens forms critical computations. Isolated, they carry insufficient information to recover the answer. Like bridge piers—the bridge collapses without them, but a pier standing alone is useless.

High-sufficiency tokens are self-contained. They carry dense information and, alone, let the model recover much of the answer signal. Like an independent tool—no other tools needed to get the job done.

These two token types play completely different roles in CoT:

• High-necessity tokens are "connective reasoning"—A implies B, B implies C; intermediate steps cannot be skipped
• High-sufficiency tokens are "key insights"—an intermediate conclusion that already contains most of the answer's information

Using only one axis fails:

• Necessity only: under aggressive compression (few tokens kept), selected tokens depend on each other and cannot individually recover the answer
• Sufficiency only: under mild compression (many tokens kept), selected tokens are all "independent insights" but lack connective reasoning—the chain breaks

MIST combines both axes: a unified necessity × sufficiency score ranks tokens stably across all compression budgets.

Engineering Breakthrough: From O(T) to Two Backward Passes

Directly computing necessity and sufficiency for every token requires an independent intervention per token—erase its residual contribution, run a forward pass, observe the change in answer log-likelihood. For a 500-token chain, that's 500 forward passes—unacceptable.

The paper's engineering contribution: both axes admit first-order Taylor linearization.

Necessity can be approximated as the inner product of the token's residual contribution vector with the gradient of the answer log-likelihood with respect to that layer's residual.

Sufficiency can be approximated as the inner product with the gradient computed on a no-chain forward pass.

Each axis requires one backward pass to score all tokens. Cost drops from O(T) to O(1)—regardless of chain length, only two backward passes are needed.

The linearization (Appendix D) is rigorous: under regularity assumptions, the Taylor remainder is bounded, and empirically tight (Appendix D.5 verifies rank consistency between the approximation and exact interventions).

Experimental Results

Across 4 reasoning benchmarks (GSM8K, MATH, MMLU-Pro, BBH-MC) and 4 models:

• MIST outperforms all baselines at every compression budget
• Baselines include TokenSkip (external scorer), Perplexity, Attention rollout, H2O (Heavy-Hitter Oracle), Uniform sampling, and No-chain
• At 30% retention, MIST significantly beats TokenSkip on MATH
• MIST generalizes well to non-math reasoning (MMLU-Pro, BBH-MC)

Ablations confirm both axes are indispensable—removing either significantly hurts performance.

Another Entry in the Current-Carrier Atlas

This paper precisely hits the "current-carrier atlas" idea—moving from "has influence" to "which pathway actually carries the current."

Previous cases:

• SCIT: latent-thinking reasoning lives in the value-cache suffix, not hidden states
• TwinKV: attention weights and causal contribution are nearly uncorrelated (ρ=-0.004); mainstream KV eviction methods rest on a flawed assumption

MIST adds another:

• Ripples in the residual stream are the true current-carrying pathway from token to answer computation
• Attention weights are not—high-attention tokens don't necessarily leave large ripples
• External scorer judgments are not—an external scorer's judgment can diverge entirely from the model's internal computation path

TwinKV found "attention ≠ causal contribution" (ρ=-0.004); MIST found "necessity ≉ sufficiency" (ρ=-0.07). Both point to the same conclusion: importance inside an LLM is not a scalar but a vector. Any method ranking tokens on a single dimension will fail in some regimes.

This reinforces the "scalar illusion"—managing a vector with a scalar is like taking blood pressure with a thermometer.

Another Instance of Judgment-Gate Decoupling

MIST can also be seen as another instance of "judgment-gate decoupling":

• Judgment module: the model internally "knows" which tokens matter (ripple size in the residual stream)
• Action module: external compression methods (e.g., TokenSkip) make deletion decisions based on external scorers
• Decoupling: the external scorer's judgment and the model's internal judgment never connect

The fix follows the same logic: don't have an external scorer guess what the model thinks is important—read it directly from inside the model. MIST's necessity score is the model's own "vote" on each token's importance.

Deeper Reflections on CoT Compression

This paper made me rethink a question I hadn't pinned down: why does CoT work at all?

If CoT were merely "reasoning unfolded into explicit steps," compression should be easy—keep the key steps, delete the rest. But MIST's finding that necessity and sufficiency are nearly uncorrelated means CoT's structure is far more complex than a "sequence of steps."

CoT is not a line but a network. Some tokens are load-bearing nodes (high necessity, low sufficiency), present so that other tokens' computations work. Some are independent insights (high sufficiency, low necessity), which alone restore much of the answer's signal.

Compressing CoT is not "deleting redundant steps" but "balancing density and completeness in a network." MIST's two-axis score approximates exactly this network structure.

Limitations and Open Questions

The paper honestly acknowledges several limitations:

1. Model scale and gradient access: MIST requires gradients and residual-stream access; it doesn't apply to closed-source API models 2. Evaluation domains: validated mainly on math and reasoning benchmarks; other domains (code, long text) remain untested 3. Approximate interventions: Taylor linearization is an approximation and may be inaccurate in extreme cases (very large or very small residual contributions)

But the deepest open question: is the necessity–sufficiency separation an essential structure of CoT, or an artifact of current training paradigms? If future models learn more compact reasoning, might this separation vanish?

A Deeper Analogy

The "ripple" in the paper's title suggests an analogy. In physics, measuring an object's effect on a field can be done two ways:

1. Remove it and see how much the field changes—this is "necessity" 2. Place only it and see how much the field recovers—this is "sufficiency"

These two measurements are equivalent in physics (linear superposition). But in an LLM's residual stream, they are not—because of the Transformer's nonlinearity.

Nonlinearity is the source of the separation. If the residual stream were linear, necessity and sufficiency would be fully equivalent. But each Transformer layer contains attention (nonlinear) and MLPs (nonlinear), making composition unequal to summation.

MIST's two-axis score essentially quantifies this nonlinearity: the larger the gap between necessity and sufficiency, the deeper the token participates in nonlinear computation.

This view makes MIST more than a compression tool—it is a probe into the computational structure of CoT.

---

Paper: Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression

Authors: Tianyi Zhao, Yinhan He, Wendy Zheng, Chen Chen (University of Virginia)

Published: 2026-08-31

Method: MIST (Model-Internal Saliency for Token-level CoT compression)

Key numbers: Necessity–sufficiency Spearman ρ=-0.07, top-30% overlap 0.28; MIST beats all baselines across 4 benchmarks × 4 models