The Puzzle: Same Model, Same Prompt, Different Answers
Run the same math problem through the same LLM ten times: six correct, four wrong. The model hasn't changed, the prompt hasn't changed — so where does the variance come from?
This isn't anecdotal. Qwen3-8B scores around 60% pass@1 on GSM1K, but nearly 100% pass@64. The capability exists; single attempts just frequently squander it. Prior research examined which *step* goes wrong, or which token is wrong *after the fact* — but nobody had pinpointed the exact token that pushes a model from "still salvageable" to "beyond recovery."
A 2026 paper — *Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning* — by Jaeyong Ko and Pilsung Kang (Seoul National University) and Yukyung Lee (Boston University) names this token the cliff token.
What Is a Cliff Token?
For every token position \(t\) in a reasoning chain, define the token-wise potential: the fraction of 64 continuations from that position that reach the correct answer. A cliff token is a position where this potential drops statistically significantly — the previous token is still recoverable, this one is not.
Why Not a Fixed Threshold?
The intuitive approach — flag any drop over 0.2 — fails because potential is estimated from 64 rollouts and has variance: small near 0 or 1, largest near 0.5 (like coin-flip noise). A fixed threshold would misclassify sampling noise as a cliff in the mid-range.
The paper instead uses an adaptive threshold:
where \(\text{SE}_t\) is the standard error of the difference in proportions and 1.645 is the 95% confidence z-score. The threshold sits around 0.18 at extreme potentials and rises to ~0.24 mid-range — raising the bar where noise is high, lowering it where noise is low. The methodological principle: distinguish "actually collapsed" from "looks collapsed but is measurement noise."
Deleting One Token Saves the Whole Chain
The causal experiment is elegantly simple. Take a wrong reasoning chain, locate the first cliff token, then:
- Cliff-del: resample from before the cliff (delete the cliff token)
- Cliff-keep: resample from after the cliff (keep the cliff token)
- Deterministic cliffs are scale-invariant: of 46 deterministic cliff positions found in Qwen3-8B, 44 were matched by Qwen3-0.6B sampling the identical cliff token; all 37 of 0.6B's deterministic cliffs were reproduced by 8B.
- Uncertain cliffs are model-specific: cross-model transfer drops probability mass by ~0.13 on average.
- Sampled-off cliffs are scale-asymmetric: 8B's sampled-off cliffs tend to be high-probability cliffs in 0.6B.
- Single-token supervision works: preference optimization at one token position can improve reasoning, opening the door to token-level training signals.
- Failure taxonomy guides treatment: sampled-off cliffs → tune temperature/top-p; uncertain cliffs → DPO/RLHF; deterministic cliffs → requires revisiting pretraining data or architecture. Diagnose before treating.
- Adaptive thresholds beat hard cutoffs in any token-level analysis (activations, attention, logits alike).
- Cross-scale diagnosis reveals family bias: if 0.6B and 8B fail identically, scaling up won't help — switch families.
- The cost: estimating token-wise potential consumed 4,047 A100-80GB GPU-hours. Cliff analysis is currently offline-only; only the Cliff-DPO-trained model deploys online.
- Only "lost potential" cases analyzed. Problems the model can't solve at all (potential near 0 from the start) produce no cliffs — on AIME 2025, 53 of 60 Llama error chains had zero potential from the start; that's capability, not cliffs.
- Math reasoning only: code generation, planning, and tool use are untested.
- N=64 rollouts is a budget compromise; the adaptive threshold partially compensates.
- Paper: Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning
- Code: github.com/beaver-22/Cliff-token
- Authors: Jaeyong Ko, Pilsung Kang (Seoul National University); Yukyung Lee (Boston University)
| Metric | Cliff-del | Cliff-keep | |---|---|---| | pass@64 | 1.00 | 0.71–1.00 |
Deleting that one token guarantees a correct answer within 64 samples; keeping it leaves ~30% of cases unrecoverable even at 64 samples. One token decides survival — this is causation, not correlation.
Three Ways to Fall Off the Cliff
Using token entropy and greedy-sampling status, cliffs split into three types:
1. Deterministic cliffs
Greedy token with very low entropy (\(H < 0.0561\) nats, greedy probability \(p_1 > 0.99\)). The model isn't hesitating — it is confidently wrong. Cliff probability mass approaches 1.0. Like walking a ridge with firm, confident strides — straight off the edge.2. Uncertain cliffs
Greedy token, but high entropy (\(H \geq 0.0561\)). Mean cliff probability mass 0.68, with wide spread (IQR 0.44–0.95). The model leans toward the cliff token but knows it's a gamble. It hesitated at the fork — then chose the wrong path anyway.3. Sampled-off cliffs
Non-greedy token with high entropy. The cliff token had mean probability mass of only 0.32 — a low-probability candidate that random sampling happened to select. A loose pebble, not a wrong turn.The three distributions (paper Figure 4) are cleanly separated: the taxonomy reflects genuine structure in the data, not arbitrary cut points.
Cross-Scale Transfer: Small and Large Models Fall at the Same Cliff
The most unsettling finding. Using cliff positions found in Qwen3-8B to examine Qwen3-0.6B (and vice versa):
Deterministic cliff invariance means these failures are not an accident of scale but a property of the model family — likely shared pretraining patterns or architectural inductive biases that post-training cannot easily erase.
Cliff-DPO: Training on the Cliff Edge
If cliff tokens trigger failure, why not apply preference optimization specifically there?
1. Find 2,926 cliff positions on the GSM8K training set 2. At each position, sample the top-10 candidate tokens and estimate each one's potential via 64 rollouts 3. Pair non-cliff candidates (chosen) against cliff tokens (rejected): 19,227 preference pairs 4. Apply DPO loss only at cliff positions
Results (Qwen3-0.6B):
| Method | GSM1K | MATH500 | AIME 2025 | Tokens updated | |---|---|---|---|---| | Baseline | 57.0 | 51.6 | 3.5 | — | | DPO (full sequence) | 56.5 | 51.0 | 2.2 | 2,862,845 | | cDPO (full sequence) | 61.3 | 54.9 | 4.8 | 5,829,052 | | Cliff-DPO (deterministic) | 57.0 | 51.5 | 2.9 | 5,538 | | Cliff-DPO (uncertain) | 62.9 | 53.3 | 3.8 | 18,122 | | Cliff-DPO (sampled-off) | 62.5 | 52.6 | 3.5 | 14,794 | | Cliff-DPO (uncertain + sampled-off) | 63.6 | 56.0 | 4.9 | 32,916 |
Key observations:
1. Uncertain + sampled-off performs best (+6.6 GSM1K, +4.4 MATH500), matching or beating full-sequence cDPO. 2. Deterministic cliff training doesn't work — consistent with them being pretraining biases outside DPO's reach. 3. Extraordinary efficiency: 32,916 token updates vs. cDPO's 5,829,052 — 177x fewer with equal or better results. Most DPO training signal is wasted on positions that don't matter.
Engineering Takeaways
Limitations
Closing Thought
Having potential (pass@k) and reliably reaching the answer (pass@1) are separated by cliff tokens — each one a stumble between "knowing" and "doing." The taxonomy says stumbles have three causes: overconfidence, hesitation, or bad luck — each needing a different fix. The scale-invariance of deterministic cliffs warns that scale is not a universal remedy. And Cliff-DPO's 177x efficiency gain suggests we may have been training at the wrong granularity all along: LLM computation — and failure — is highly sparse, and the effective signal lives in a handful of positions.