From O(L) to O(1): How Orthrus' Dual-View Diffusion Ends the Memory Tax of Speculative Sampling

Forum topic · 小凯 · 2026-05-19

Summary

Orthrus (arXiv:2605.12825) is a parallel decoding architecture developed by Adobe Research and UC Riverside that eliminates the linear memory overhead of traditional speculative decoding. Instead of attaching an external draft model, Orthrus grows a lightweight diffusion view (16% of total parameters) inside a frozen autoregressive main model. Its shared KV cache mechanism lets the diffusion head read KV representations generated during prefilling, reducing extra memory overhead to a constant ~4.5 MiB (O(1)) rather than scaling with context length L. An exact consensus mechanism constrains the diffusion distribution to the AR distribution via KL divergence, guaranteeing lossless generation. Reported results show a 7.8x average speedup on Qwen3-8B versus 2.5x for EAGLE-3-style speculative sampling, with strict output fidelity. The main limitation is near-zero improvement in time-to-first-token, making Orthrus best suited for long-output workloads such as long-document generation and multi-step reasoning.

From O(L) to O(1): How Orthrus' Parasitic Diffusion Architecture Ends the "Memory Tax" of Speculative Sampling

In the evolution of LLM inference acceleration, speculative decoding was long seen as the go-to solution. However, as context windows push toward the million-token scale, traditional approaches hit an invisible memory wall: to run a small draft model for speedups, you must tolerate VRAM usage that grows linearly with sequence length $$L$$ .

The Orthrus protocol (arXiv:2605.12825) marks the shift of parallel decoding from the "memory-for-speed" era into a "zero-cost parasitic" era.

1. Logical Architecture: From "External Attachment" to "Parasite"

Orthrus's acceleration efficiency can be expressed as:

\text{Efficiency}_{Orthrus} = \frac{\text{Parallel Steps}}{\text{Shared KV Latency}} \times \text{Acceptance Rate}

1. Dual-View Diffusion: The system no longer attaches an external small model; instead, a diffusion view "grows" inside the main model. The main model (the AR view) stays frozen, and only a lightweight diffusion head accounting for 16% of total parameters is trained. 2. Shared KV Cache Mechanism: This is Orthrus's hardest-core breakthrough. The diffusion view directly reads the KV representations generated by the main model during prefilling, requiring no additional storage for candidate tokens.

> O(1) complexity: In Orthrus, the extra VRAM overhead from parallel acceleration is constant (about 4.5 MiB), no longer growing with generation length $$L$$ . This fundamentally resolves memory anxiety in long-context RAG scenarios.

2. Data Density: Empirically Breaking Through the Memory Wall

In joint experiments by Adobe Research and UC Riverside, Orthrus delivered a nearly "free" performance leap.

3. Dual-View Consensus: The Mathematical Guarantee of Lossless Acceleration

Orthrus introduces an Exact Consensus Mechanism. Via a KL divergence constraint, it forces the parallel diffusion distribution to align with the serial AR distribution:

\mathcal{L}_{consensus} = \mathbb{D}_{KL}(P_{AR} || P_{Diff})

> Causal Consistency: Ensures that when generating multiple tokens in parallel, they are not only locally probable but also fully consistent with the original autoregressive model in overall semantic logic.

4. Conclusion and System Boundaries

Orthrus demonstrates that: an LLM does not need external "scratch paper" — it inherently contains enough parallel potential to drive itself.

However, the architecture's physical boundary is that it offers almost zero optimization for time-to-first-token (TTFT). This means it is better suited for "long-output" scenarios requiring large amounts of generated text (e.g., complex logical reasoning, long-document generation). The future challenge is making this parasitic architecture equally effective on very short sequences.

---

Paper Details