NgramRL: AI Starts Copying Its Own Homework—And It's Perfectly Justified
Imagine a final exam where the last question is identical to last week's homework. The rules are strict: no notes, show every step. You remember the answer, but you must re-derive it—only much faster this time, checking each line mentally against the draft spread beside you. The submitted exam is indistinguishable from one derived from scratch, but it took half the time.
That's essentially what researchers at UC San Diego and West Virginia University built in October 2026. They let a large language model undergoing reinforcement learning (RL) training openly "consult its own homework." The method is called NgramRL (arXiv: 2610.32881). It never peeks at external answers, doesn't change the trained model, and doesn't alter any property of the learning process—yet it speeds up the most expensive part of RL post-training by nearly 1.4x.
The interesting part isn't the speedup itself; it's the fact it exposes: during training, AI has been endlessly repeating itself while pretending every time is the first.
🧾 An Overlooked Bill: Where the Money Goes
The mainstream way to improve LLM reasoning is RL post-training, exemplified by GRPO: give the model a batch of problems, have it write multiple answers per problem (rollouts), compute rewards based on correctness, and adjust parameters accordingly.
Here's the cost structure: most people assume training is expensive because of backpropagation. But in RL post-training, the dominant cost is rollout—generating answers token by token. The paper reports rollouts consume over 70% of training time.
Why? Generation is autoregressive: each token requires a full forward pass through billions of parameters. And GRPO makes it worse—it samples each prompt multiple times (5 completions per prompt, 64 prompts per step, across 15 epochs in the paper's setup). This repetition isn't engineering laziness; it's an algorithmic requirement. GRPO compares answers *within a group*—relative performance, not absolute scores. Too few samples means the comparison is noise.
So RL post-training essentially replays "generation" over and over inside the training loop. To save money, either generate less (touching the algorithm's core) or make each generation faster. NgramRL takes the second path.
⚡ Speculative Decoding: Drafts Run First, the Examiner Verifies
The classic acceleration idea is speculative decoding, resolving a tension: large models are accurate but slow; small models are fast but inaccurate.
A small draft model first guesses k tokens in one go. Then the large model verifies all k in a single parallel forward pass. Accepted tokens pass; at the first rejection, generation truncates and the large model samples its own token from a residual distribution.
The acceptance rule: a draft token is accepted with probability min(1, π_θ(x|c)/q(x|c)), where π_θ is the current policy and q is the draft. The output distribution is mathematically identical to sampling from the large model alone—lossless in quality, just faster.
Example: if the draft's confidence is q=0.9 and the big model's is π=0.95, acceptance probability is 1. If the draft says q=0.9 but the model says π=0.1, the ratio is ~0.11—rejected nine times out of ten, with resampling almost certainly from the model's preferred token.
It's like a senior partner reviewing an intern's contract draft: pages that pass go through quickly; problematic clauses get rewritten on the spot. The delivered document is legally identical to one written entirely by the partner—at a fraction of the partner's time.
So why does this break down in training?
🧨 Why Old Methods Fail in the Training Arena
Model-based drafting (EAGLE, Medusa). These train a separate small draft model. Two problems: (1) it occupies GPU VRAM, which is already packed with gradients, optimizer states, and activations; (2) worse, the policy model changes every step during training—your draft model goes stale within days, mismatch rises, rejection rates climb. A stale draft is worse than no draft.
Model-free drafting (prompt-lookup, REST). These look up candidate tokens from the current prompt or a static corpus. Great for "restatement" tasks like summarization where the answer hides in the input. But reasoning rollouts—math derivations, code, database queries—barely overlap with the prompt. You can't find a proof inside the problem statement.
Both routes point to the same missing ingredient: a draft source that understands the problem, occupies no VRAM, and tracks the evolving policy. Understanding requires capability (rules out static corpora); no VRAM rules out draft models; tracking the policy rules out anything static.
The NgramRL authors did something clever: instead of searching for such a source, they counted.
🔍 The Waste Nobody Looked At: Rollouts Repeat Themselves
They measured how much repetition exists between rollouts of the same prompt across training epochs:
- Early training: ~50% of a prompt's rollouts are identical
- Mid-training: 81%
- Late training: 100% verbatim—the model has developed muscle memory; every rollout is a token-perfect replica
- Coverage between consecutive epochs: 48%–96%
- Repeated fragment lengths: 7–23 tokens on average
- NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding
This is structural repetition, not noise. GRPO resamples the same prompts, while each gradient step only nudges the policy slightly—so its answers to a given problem barely change. The repetition curve itself mirrors training progress: exploration early, consolidation mid, convergence late.
And there's the answer: the rollouts themselves are the ideal draft source. They're produced by the policy, so they track it automatically—no staleness. They live in CPU memory—no VRAM cost. And their knowledge of each problem is "I did it this way last time"—more intimate than any external corpus.
Back to the exam metaphor: the smart student doesn't borrow a classmate's notes (model-based drafting) or dig through the textbook (model-free). He spreads his own last week's homework on the desk corner as scratch paper.
📓 NgramRL: A Living Notebook per Problem
The core data structure is almost embarrassingly simple: an n-gram lookup table.
NgramRL chops past rollouts into fragments of at most 6 tokens (n≤6), stored in a hash-style table. During generation, the current suffix serves as the key; a match yields a draft proposing the next k=16 tokens. Halfway through writing an answer ending in "so the final answer is," the table reveals the historical rollout continued "42, because..."—one forward pass verifies the whole string at once.
Three key design decisions:
1. One table per prompt, no global sharing. Counterintuitive, but the ablation is clear: a global cache achieves 2.41 tokens/forward vs 3.05 for per-prompt. Even 4x global capacity only reaches 2.65. Repetition clusters *by problem*—mixing different problems' homework in one drawer just adds clutter.
2. Tables live in CPU memory. Zero GPU VRAM impact. No luggage added to the already-packed train.
3. Tables are alive, continuously updated with rollouts from the most recent W steps. Policy moves, the table follows. Old rollouts get evicted; new ones take over.
How big should W be? W=4 captures 79–91% of the benefit; beyond W=16, performance *drops*—stale rollouts mismatch the current policy and waste verification compute. You only need last week's homework; last semester's notes have limited value.
👨🏫 Why Copying Isn't Cheating
Legitimate concern: does referencing its own outputs pollute training? Could the model collapse into parroting itself?
NgramRL's answer is mathematical, not hopeful. It uses the standard speculative sampling correction: drafts propose, the current policy judges with probability min(1, π_θ/q). The more the draft agrees with the policy at a position, the more readily it's accepted; the wilder it is, the more likely it's rejected and replaced by a sample from the policy's own residual distribution. The pipeline's output distribution is strictly identical to unhurried token-by-token sampling from the current policy.
This cleanly separates "copying" from "learning." Drafts provide speed; the policy provides verdicts. Every token used for rewards and gradients comes from the current policy's distribution. Training dynamics, rewards, gradients, and the final model are all unchanged. Pure acceleration, zero side effects.
This matters more in RL than in inference because RL learns *from* the accelerated data. If acceleration secretly shifted the distribution, it would alter the raw material of learning—reward statistics, gradient directions, convergence trajectories—all changed, producing a different model. NgramRL puts losslessness at the center: write "don't alter answers" into the school rules before discussing speed.
🚰 The Real Bottleneck Isn't the Algorithm—It's the Pipeline
In theory, the story ends here. In practice, the authors hit a wall every paper design hits: the wall between CPU and GPU.
The cache is in CPU memory; the model is on GPU. Every decode step requires a round trip: GPU sends context to CPU → CPU looks up → drafts return to GPU. Measured result: the GPU idled ~29% of the time. Not because lookup is slow, but because of synchronization and scheduling overhead—like a coffee machine stopping after every cup to wait for a slow grinder.
NgramRL's systems answer is its most elegant contribution: the Adaptive Fan-out Prefetcher.
The idea in one sentence: don't wait for the GPU to request drafts—prepare the next batch while the GPU is still verifying the previous one.
Each verification step produces a result (j, b): j tokens accepted, draft guessed b at position j+1. Before the result arrives, the CPU can't know (j, b)—but it can maintain a causal sliding-window estimate of historical P(j), plus a per-branch coverage profile. Then it greedily allocates a prefetch budget of N=16 candidates to the (j, b) combos with the highest expected coverage.
One beautiful observation: when all draft tokens are accepted, the next token is the continuation in the cached trajectory—and it's the top-1 candidate ~90% of the time. So the all-accept branch needs only f_j=1 prefetch slot, nearly never wasted.
Results: on Spider, prefetch coverage hit 96.6% (Qwen3-8B) and 99.5% (DeepSeek-7B). End-to-end speedup: 1.38x against a 1.40x theoretical ceiling—98.6% of maximum. Occasional misses fall back to synchronous lookup; committed tokens are never touched.
📊 The Numbers
Generation throughput. With a warm cache, NgramRL achieves 3.60–7.77 tokens per forward pass—up to 1.99x over SuffixDecoding and 7.0x over prompt-lookup.
Ablations confirm the foundation. Global 2.41 vs per-prompt 3.05 tokens/forward (global at 4x capacity: 2.65; SuffixDecoding: 2.43).
Heavy-tailed distributions hide gold. On Spider, 14.1% of steps accepted 12+ tokens, and 10.1% accepted all 16—verbatim replay of entire passages. Other tasks: 13.0% (DeepSeek on GSM8K), 6.4% (Qwen on GSM8K), 3.7% (CodeContests).
Two telling dynamics. On Spider, acceptance peaks mid-training then declines as outputs diversify and the cache ages—proof the table is alive. On CodeContests, a reversal: the global cache led early (shared code templates across prompts) before per-prompt caught up.
Setup. 2× A100 80GB, Qwen3-8B and DeepSeek-LLM-7B, on GSM8K, Spider 1.0, and CodeContests; 64 prompts × 5 completions × 15 epochs per step.
🧠 A Deeper Look: The Archaeology of Waste
Training-time waste is uniquely insidious. Inference waste is visible—extra tokens are extra dollars on the bill. Training waste hides under the banner of "learning": nobody questions that sampling is part of training. But the numbers (50%, 81%, 100%) reveal that much of it isn't sampling at all—it's replay, and replay doesn't need recomputation.
NgramRL effectively turns the model's own history into an externalized working memory. This is the first time learning's products feed back into learning's process—not via distillation or experience replay, but through direct alignment at the generation layer. Could cached trajectories also serve as probes or diagnostic tools, exposing the policy's confidence structure? The paper doesn't say—but it's the most interesting question it leaves behind.
And one engineering lesson worth memorizing: the bulk of the final gain came from a pure systems component—the prefetcher. Between "can accelerate in theory" and "actually accelerates" lies a CPU-GPU conveyor belt. In today's AI systems research, the deciding factor often lives not in the model, but in the seams exposed by 29% idle time.
AI has learned to copy its own homework—and it does so with full justification, because it re-verified every page. The real old ledger it uncovered: we made AI do its most repetitive work in the most expensive way for too long. And the answer was always written in its own notebook.