Prefix Sliding Fact-Check: 'The Longer You Think, the Pricier It Gets' Is Fixed — But the 'Think-and-Forget' Intuition Maps to the Baseline That Lost

1. The Verdict Table

| Video claim | Paper text | Verdict | |---|---|---| | Slow-thinking models like o1 and DeepSeek-R1 need Full Attention | "keep the entire reasoning trace in memory via full attention, as used in most language models" (Sadhukhan et al. 2025) | Reasonable example: the paper says "most models," names neither, but the example direction is correct | | Loops, endless repetition, OOM | Paper lists four long-context ailments: distraction / context poisoning / repetitive loops (Pipis et al. 2025) / lost knowledge, plus linear cost growth | Accurate, faithful listing | | "Top teams like Stanford" | Muennighoff, Ludwig Schmidt, Percy Liang, Andrew Ng (Stanford) + UW (Zettlemoyer / Yejin Choi / Mike Lewis) + UCSB + Prime Intellect, 17 authors | Accurate, all-star lineup (first author Muennighoff is a veteran of s1/simple-evals) | | Math analogy (problem = memorized prefix / scratch work = discarded / current line = window) | Paper's example ((42+84)×4)−5: once 42+84 is computed, the intermediate reasoning is no longer needed, only the result matters | Faithful, analogy maps sentence by sentence | | "Cost becomes constant" | "The cost of generating an additional token is the same regardless of whether the model has already generated millions or billions of tokens" | Accurate, near-direct quote — but with a warm-up precondition (see caveat ①) | | "Memory stays rock stable" | Table 1: Full throughput falls 1477→448 tok/s from 32K→128K (3.3x decay); window 4096 stays flat at 5479→5224 | Accurate with engineering evidence (custom Hopper/FlashAttention kernels) | | 3x speedup | "3x faster while maintaining performance" (training-free version); 5479 vs 1477 at 32K ≈ 3.7x | Accurate, gap widens with longer generation |

2. The Mismatch: 'Think-and-Forget' — Summary Lost to 'Just Drop It'

The paper's control design is more informative than the method itself. Three constant-memory schemes compete:

• Last k: generate to threshold n, keep only the last k tokens (k=256). Already widely used in the agent world (dropping old turns in multi-turn chat; the theoretical version is Markovian Thinking/Delethink). Failure cause: shifting context windows mean the last-k chunk gets processed twice; large k makes recomputation expensive, small k loses in-use information, and memory oscillates.
• Summary: generate to threshold, summarize everything, restart the window carrying only the summary — this is the strict implementation of "think-and-forget." Failure cause: models lose important information across successive summarization rounds (Wang et al. 2026a), and it adds a pile of hyperparameters (summary length, summarizer, trigger threshold).
• Prefix Sliding: full prefix kept + a 4096-token sliding window of recent tokens; everything in between is dropped with no substitute.

Result: Prefix Sliding wins on AIME25. The ordering itself is a cognitive update — compressing intermediate thinking yields less than simply discarding it, provided the prefix and the current window remain. One mechanism detail the video never mentions: the second reason the prefix must be kept is the attention sink — the prefix acts as an attention basin absorbing excess probability mass; lose it and the model cannot even read "now" stably. So the recipe is a two-part structure of "anchor + window," both indispensable.

As for the performance-parity mechanism, the paper is honest: "Prefix Sliding performs better as it can generate more tokens in the same thinking time than full attention, not because each token it generates is better" — the speedup comes from throughput, not intelligence; the 3x thinking budget buys search breadth.

3. The AIME Numbers: Window Is a Recipe, Not a Switch

Table 1 (Qwen3-1.7B, avg@64):

| Window | AIME25 | Avg length | Throughput (32K→128K) | |---|---|---|---| | 2048 | 27.7 (down 6.5) | 47643 | 8973→8737 | | 4096 | 33.9 | 29943 | 5479→5224 | | 8192 | 35.8 (beats Full) | 19373 | 3291→2788 | | Full | 34.2 | 19158 | 1477→448 |

Three observations: ① too small a window is lossy (2048 drops 6.5 points) — "maintaining performance" presupposes window ≥ 4096; ② window 8192 beats Full — because it generates more tokens in the same wall-clock time; ③ bigger windows are more stable but slower — this is a mixing curve, not a switch.

4. The Bigger Picture: Three Ways to Throw Away the Scratch Paper

"CoT verbosity is essentially working memory outsourced through the narrow gate of discrete tokens" — the ledger now has three layers of solutions, all appearing within the past two weeks:

1. Latent-space level (Mobius): swap the scratch paper for RAM — reasoning moves into latent space, no tokens written at all 2. Symbolic level (Prime Agent context-as-variables): scratch work moves from the token stream into variables — what enters the token stream becomes an explicit programming decision 3. Discard level (Prefix Sliding): accept that reasoning runs through the token stream, but give the stream a garbage collector — intermediate tokens dropped by staleness

All three share the same physics: the token stream is an expensive interface, and scratch work must avoid occupying it. Prefix Sliding adds the in-flight accounting dimension — Prime Agent covered "what enters the token stream" (ex ante), this covers "when it expires after entering" (in flight): most intermediate reasoning tokens lose importance, on a timescale roughly equal to the window size.

Second connection: layering within the token stream = the reasoning-side version of ticket/fog criteria. Prefix = task definition (persistent, precisely stateable structure — the ticket); intermediate thinking = work in progress (not pre-stateable, discardable — the fog); sliding window = the executing present. The criterion is isomorphic: "can it be stated precisely right now" decides whether it enters the prefix or the discard zone.

Third connection: Prime Intellect's third appearance — after prime-agent and the RLM lineage, this time prime-rl serves as the async framework for the paper's RL experiments (GRPO via trl synchronous / prime-rl asynchronous). Decentralized RL labs' infrastructure is becoming the default for frontier academic groups — a trend worth its own file.

Fourth connection: the training-side increment the video skipped — in RL training, "truncating and discarding overlong generations" is industry practice (Yu et al. 2025). Prefix Sliding makes complete >100k-token rollouts affordable, and backward passes from truncation only need the last sliding window (passing 4× window tokens guarantees gradient accuracy, validated by KL). This aligns with RLM-Qwen3-8B's "train the model for structure": cache policy sinking from harness decision to trainable structure.

5. Honest Boundaries (None of Which the Video Covered)

① LiveCodeBench regression = the counterexample where scratch work is a half-finished product: the model writes a function mid-reasoning, thinks for thousands of tokens in comments, and when it returns the function's opening has slid out of window — "intermediate tokens lose importance" has a structural exception on code tasks (the draft is not waste paper, it is an unfinished product). The paper offers two remedies: RL training so the model writes shorter comments, or a mechanism letting the model append tokens into the prefix (the latter is an architecture direction worth watching). ② Zero benefit on short tasks: when generation length < window, it is just full attention (warm-up phase); HealthBench tasks averaging 2086 tokens get essentially no speedup. ③ Agentic flooding risk: reading web pages/files can fill the window — when the window is smaller than the content it strictly cannot read it all, and new output flushes content still in use. How later user instructions in multi-turn conversations are handled is also unsolved. ④ Experimental base is Qwen3-1.7B: small model first; validation on frontier large models is missing (figures include 8192/16384 windows, no >7B results seen). ⑤ Baselines admittedly limited: only "out-of-the-box + constant memory" schemes compared; H₂O-style cache eviction relegated to the appendix. ⑥ 22 stars, released a week ago — propagation just starting, no third-party reproduction yet.

6. A Falsifiable Prediction

Within 12 months, vLLM/SGLang will make prefix+sliding KV modes a first-class switch (the Hopper kernel is already hand-written, so the engineering distance is short), or some frontier lab will ship a flagship "Prefix-Sliding-trained, million-token inference" model (a new inference-time scaling selling point). Either happening validates it; if mainstream inference frameworks still only offer naive sliding windows a year from now, then LiveCodeBench-type counterexamples hold broadly at large scale, and the applicable domain of scratch-paper GC is narrower than the paper claims.

---

*Verification notes: arXiv 2608.26070v1 (HTML full text, 92K characters, captured 2026-09-05); github.com/Muennighoff/prefix-sliding (22★, push 08-27); Table 1 and the four Limitations checked section by section. Of the video's seven claims: five accurate, one reasonable example, one flagged mismatch between the 'think-and-forget' intuition and the actual mechanism (the mismatch itself is more informative than an error — the intuitive scheme, Summary, measured worse).*