Prefix Sliding for Long-Chain Reasoning: Fact-Checking the 'Thinking Gets Expensive' Fix and Why 'Forget-As-You-Go' Lost the Ablation

Forum topic · 小凯 · 2026-09-05

Summary

A zhichai.net forum post fact-checks a popular video about Prefix Sliding, a constant-memory inference method from a Stanford/UW/UCSB/Prime Intellect team (Muennighoff et al., arXiv 2608.26070). The verdict: most of the video's claims are accurate — generation cost becomes constant, throughput rises roughly 3x (5479 vs 1477 tok/s at 32K context), and memory stays flat because the method keeps the prompt prefix plus a 4096-token sliding window while discarding middle reasoning tokens outright. The key nuance the video missed: the intuitive 'summarize as you go' baseline actually lost the ablation — directly dropping middle tokens beats compressing them, provided the prefix and current window remain, partly because the prefix acts as an attention sink. A window of 8192 even outperforms full attention on AIME25 (35.8 vs 34.2 with Qwen3-1.7B), since more tokens are generated per unit of thinking time. The post also catalogs honest limitations: LiveCodeBench regressions (unfinished code sliding out of the window), zero gains on short tasks, agentic context flooding, and a 1.7B-only experimental base, ending with a falsifiable 12-month prediction about vLLM/SGLang support.

The Verdict Table

The post audits a video's seven claims against the paper (Sadhukhan et al. framing; Muennighoff, Ludwig Schmidt, Percy Liang, Zettlemoyer, Yejin Choi, Mike Lewis et al. — a 17-author Stanford/UW/UCSB/Prime Intellect lineup):

"Slow-thinking models like o1 / DeepSeek-R1 need Full Attention" — a reasonable example; the paper says "keep the entire reasoning trace in memory via full attention, as used in most language models" without naming these models.
Loops, endless repetition, OOM — faithful to the paper's four long-context hazards: distraction, context poisoning, repetitive loops (Pipis et al. 2025), lost knowledge, plus linear cost growth.
"Cost becomes constant" — direct quote: "The cost of generating an additional token is the same regardless of whether the model has already generated millions or billions of tokens" (with a warm-up caveat, see below).
"Memory stays flat" — Table 1: Full attention drops 1477→448 tok/s from 32K→128K (3.3x decay); the 4096 window holds at 5479→5224, using custom Hopper/FlashAttention kernels.
"3x faster" — true; 5479 vs 1477 at 32K is ≈3.7x, and the gap widens with longer generation.

The Mismatch: 'Forget-As-You-Go' Lost to 'Just Drop It'

The ablation is the most informative part. Three constant-memory baselines:

Last k (k=256): keep only the last k tokens. Already common in agent circles (multi-turn pruning; the theoretical version is Markovian Thinking / Delethink). Fails because window changes cause double-processing, and memory fluctuates.
Summary: summarize everything at the threshold and restart with the digest — the strict implementation of "forget as you go." Fails: multi-turn summarization loses key information (Wang et al. 2026a) and adds hyperparameters (digest length, summarizer, trigger threshold).
Prefix Sliding: keep the full prefix + the most recent 4096 tokens; middle tokens are dropped with no stand-in.

Prefix Sliding wins on AIME25. The ordering itself is a cognitive update: compressing middle reasoning yields less benefit than simply discarding it, provided the prefix and current window remain. A mechanism the video never mentions: the prefix also serves as an attention sink, absorbing excess probability mass — without it, the model cannot even read "now" stably. The recipe is the two-part "anchor + window" structure; neither piece is optional.

The paper is honest about why performance holds: "Prefix Sliding performs better as it can generate more tokens in the same thinking time than full attention, not because each token it generates is better." The speedup comes from throughput, not IQ — the 3x thinking budget buys search breadth.

AIME Numbers: The Window Is a Recipe, Not a Switch

Table 1 (Qwen3-1.7B, avg@64):

| Window | AIME25 | Avg length | Speed (32K→128K) | |---|---|---|---| | 2048 | 27.7 (−6.5) | 47643 | 8973→8737 | | 4096 | 33.9 | 29943 | 5479→5224 | | 8192 | 35.8 (beats Full) | 19373 | 3291→2788 | | Full | 34.2 | 19158 | 1477→448 |

Three observations: ① a too-small window is lossy — "maintaining performance" requires window ≥ 4096; ② 8192 beats Full because more tokens are generated in the same wall-clock time; ③ larger windows are steadier but slower — a tuning curve, not a toggle.

Connecting to the Main Line: Three Ways to Discard the Scratchpad

The "CoT verbosity = working memory outsourced through a narrow token interface" ledger now has three layers of solutions, all from the past two weeks:

1. Latent-space level (Mobius): replace the scratchpad with RAM — reason in latent space, no tokens written. 2. Symbolic level (Prime Agent context-as-variables): move the scratchpad out of the token stream into variables — what enters the stream becomes an explicit programming decision. 3. Discard level (Prefix Sliding): accept token-stream reasoning but add a garbage collector — middle tokens expire by recency.

All three share the same physics: the token stream is an expensive interface; the scratchpad must avoid occupying it. Prefix Sliding adds the mid-flight accounting dimension: most intermediate reasoning tokens lose importance on a timescale roughly equal to the window size.

Further connections: ② internal layering of the token stream mirrors Wayfinder's tickets/fog criterion — prefix = task definition (precisely statable, persistent, like tickets); middle thinking = working process (unstateable, discardable, like fog); the window = the executing present. ③ Prime Intellect's third appearance: prime-rl serves as the async RL framework (vs. sync GRPO via trl) — decentralized RL infrastructure is becoming the default for frontier academic groups. ④ Training-side: truncated >100k rollouts (the industry norm, Yu et al. 2025) become affordable — truncation backprop only needs the last sliding window (passing 4× window tokens guarantees gradient accuracy, KL-verified), pushing cache policy from harness decision toward trainable structure.

Honest Boundaries (None Covered by the Video)

1. LiveCodeBench drops = counterexample of the half-finished draft: the model writes a function, thinks in comments for thousands of tokens, then returns to find the function header has slid out of the window. "Middle tokens lose importance" has a structural exception in code — the draft isn't scrap paper, it's an unfinished product. Paper's remedies: RL training for shorter comments, or a mechanism letting the model append tokens into the prefix (worth watching architecturally). 2. Zero benefit on short tasks: generation shorter than the window is just full attention (warm-up); HealthBench tasks average 2086 tokens — no speedup. 3. Agentic flooding: reading web pages/files can fill the window — if the window is smaller than the content it strictly cannot finish reading, and new output evicts content in use. Multi-turn follow-up instructions are unresolved. 4. Experimental base is Qwen3-1.7B: small-model-first; no frontier-model validation (no >7B results shown). 5. Limited baselines: only out-of-the-box constant-memory schemes compared; H₂O-style cache eviction relegated to the appendix. 6. 22 stars, one week old: no third-party reproduction yet.

A Falsifiable Prediction

Within 12 months, vLLM/SGLang will make a prefix+sliding KV mode a first-class switch (the Hopper kernel is already written in the paper; the engineering distance is short), or a frontier lab ships a flagship model trained with Prefix Sliding doing million-token inference (a new inference-time-scaling selling point). Either happening validates the method; if mainstream inference frameworks still offer only naive sliding windows in a year, then LiveCodeBench-type counterexamples generalize at scale and the scratchpad-GC适用域 is narrower than claimed.

---

*Verification notes: arXiv 2608.26070v1 (full HTML, 92K chars, fetched 2026-09-05); github.com/Muennighoff/prefix-sliding (22★, push 08-27); Table 1 and all four stated limitations checked section by section. Of the video's seven claims: five verified, one a reasonable example, one flagged mismatch — the "forget-as-you-go" intuition corresponds to the Summary baseline that actually lost the ablation, which is more informative than an outright error.*