DLite: Speculative Decoding Without a Draft Model's Own History
Start with two numbers. First: speculative decoding can accept four or five draft tokens per pass on ordinary tasks, but in the long-context settings where agents work, the best existing methods drop to just one or two — a loss of over 80%. That's the opening comparison chart in DLite, a paper posted on October 16 by Guohao Dai's group at Shanghai Jiao Tong University, in collaboration with Tsinghua and Infinigence. Second: the authors measured the multi-token prediction module of DeepSeek-V4 with SGLang and found that merely updating and maintaining the draft-side KV cache eats 10% to 20% of end-to-end inference latency. In other words, a mechanism meant to speed things up slows down and charges a toll in agent scenarios.
The Diagnosis
Where does the problem come from? The paper traces both symptoms to a single cause: the draft model is an independent model that must read and store the entire historical context itself. In ordinary chat, contexts are short and the cost is negligible. In agent scenarios, context is long and constantly evolving — every interaction round makes it longer. The history the draft model redundantly encodes grows longer and longer, its predictions get less accurate (acceptance rate collapses), and its KV cache balloons (system overhead explodes). Once the cause is clear, the fix follows naturally: the history has already been processed once by the target LLM — why should the draft model read it again? DLite directly takes the target model's already-computed hidden representations as input, turning draft prediction from "modeling an ever-longer history" into "local prediction around the current decoding position." Problem complexity no longer grows with sequence length, and the draft-side historical KV cache drops to zero.
The Architecture
The implementation has two layers. The bottom layer is a history-free parallel backbone: because the target model's hidden states come from a different distribution than ordinary token embeddings, the model learns a set of "depth embeddings" to distinguish them. Which target-model layers feed the draft also matters — first and last layers are always kept, middle layers sampled uniformly: 12 layers for Qwen3-4B, 14 for 8B. The top layer is a lightweight low-rank RNN-style sequential head (rank 320/512): parallel prediction inherently cannot see the earlier draft tokens within a speculative block, so this head recursively compresses already-generated tokens in the block into a state, restoring intra-block causality while avoiding the expensive full-vocabulary projection.
Training is two-stage: first train a structurally isomorphic teacher model with history to bridge the structural gap, then align distributions with the target model. The whole run used 32 H800 GPUs, with 1.3 million samples from Open-PerfectBlend plus 600,000 from NVIDIA Nemotron.
Results
The results land in a comparison table across three scenarios:
- General tasks: little difference between methods. On Qwen3-4B, DLite averages 6.38 accepted tokens, slightly beating the current best, DSpark, at 6.27 — everyone passes six or seven tokens at once in these comfortable settings.
- Long-context and agent tasks: this is where they diverge. On SWE-Bench, DLite achieves 2.74x speedup with 3.19 average accepted tokens, versus DSpark's 1.64x and 1.88 — that's how the 70% lead is computed.
- AgentLongBench: even more striking — the parallel drafting method DFlash drops to 0.95x, slower than not using speculative decoding at all, while DLite reaches 2.31x.
Two Corrections to Circulating Claims
1. Phrases like "first ever" or "first to systematically identify" do not appear in the paper — the original text only says "we identify the growing history as the common root of both problems and propose DLite." 2. "Code not yet released" is outdated: the repository was created on October 7, with a push on October 21, hosted under SJTU-DAI-Sys, with 47 stars; model weights are also on HuggingFace.
The Bigger Picture
Placing this chart in the broader lineage makes it more interesting. On this drafter-acceleration line: EAGLE3 is autoregressive drafting (preserves causality but serial); DFlow, DFlash, and DFlash2 are parallel drafting (fast but losing intra-block causality); DSpark is the current best; and DLite takes a more fundamental path — the draft model no longer owns its own history. The same night, the author wrote a companion post on ECHO — DeepSeek-style MTP makes the backbone's final layer serve as its own draft head, while DLite lets an external draft head consume the backbone's hidden states directly. The two approaches share an insight: since the history is already digested in the backbone, no one should carry it a second time. The DFlash2 drafter shipped with Beam's open-source release belongs to this same family. Within one week, several leading groups pushed the same judgment forward from three directions — the future of drafting lies not in smaller models, but in less history.