DiffusionGemma Deep Dive: From Token-by-Token to Whole-Block Generation
> Release: June 10, 2026, Google DeepMind, Apache 2.0 > Hugging Face: google/diffusiongemma-26B-A4B-it
TL;DR
DiffusionGemma is not a "faster autoregressive model" — it switches the generation paradigm entirely, from GPT-style token-by-token decoding to Stable Diffusion-style whole-block denoising. Using a 256-token canvas, bidirectional attention, and iterative denoising, it achieves 1,100+ tokens/s on H100 (4x a similar-size AR model), at the cost of 5-20 points on some benchmarks versus Gemma 4. It is a structural shift that turns LLM inference from memory-bound into compute-bound.
Why Autoregressive Generation Has a Speed Ceiling
All GPT/Claude/Gemma models are autoregressive (AR): each token must wait for the previous one, making decode serial.
| Problem | Cause | Consequence | |---|---|---| | Memory-bound | KV cache read/write every token | GPU compute idle, bandwidth saturated | | No parallelism | Token N depends on N-1 | 256 tokens = 256 forward passes | | Redundant attention | Recomputes attention over history | Quadratic cost with long context |
Diffusion models (like Stable Diffusion) offer an alternative: start from noise and iteratively denoise all positions in parallel. The open question was whether text — with strict sequential and grammatical dependencies — could tolerate parallel generation.
Architecture: Encoder-Decoder + Block-Autoregressive Diffusion
DiffusionGemma keeps the Gemma 4 MoE backbone and redesigns the generation head:
- Encoder: standard causal-attention AR encoder processing the prompt into a KV cache (26B MoE, 3.8B active).
- Decoder: a diffusion decoder with bidirectional (non-causal) attention, operating on a 256-token canvas with T denoising steps (10-48), processing all 256 positions in parallel each step.
- Finance dashboard generation: working code, but more small errors to fix manually than AR; code appears whole-block rather than typed line-by-line.
- Arcade game build: core logic correct, but edge handling and polish inferior to AR — diffusion's "global planning" suits architectural code, less so fine implementation.
- Measured on H100 (RunPod): ~800-1,000 tok/s for short code, ~600-800 for long documents (canvas stitching overhead), ~400-600 for multi-turn chat (context re-encoding each turn).
- Pareto front: 10 steps is fast but weak; 48 steps approaches AR quality but loses speed advantage. A true sweet spot may require larger models or better training.
- Long-document coherence: 256-token canvas boundaries may produce a "stitched" feel in papers and novels.
- UX latency perception: block output removes the token-by-token "thinking" feedback users expect.
- No prefix caching: bidirectional attention precludes vLLM-style incremental KV reuse; long multi-turn conversations may be less efficient than AR.
- Training cutoff: January 2025; no audio input (unlike Gemma 4 12B).
- Hugging Face: https://huggingface.co/google/diffusiongemma-26B-A4B-it
- Gemini Diffusion: https://deepmind.google/models/gemini-diffusion/
- vLLM recipe: https://recipes.vllm.ai/Google/diffusiongemma-26B-A4B-it
- NVIDIA NIM docs: https://docs.api.nvidia.com/nim/reference/diffusiongemma-26b-a4b-it
- ModelScope GGUF: https://modelscope.cn/models/unsloth/diffusiongemma-26B-A4B-it-GGUF
- Source video: https://www.youtube.com/watch?v=Dxn3BcSgsMY
Key design decisions
1. Block-autoregressive: canvases of 256 tokens are denoised in parallel internally; consecutive canvases are generated serially, each conditioned on the previous via the encoder's KV cache. This balances parallelism with long-range consistency. 2. Bidirectional attention (a BERT inheritance): every position sees the entire canvas, including "future" tokens. Cost: no incremental KV-cache reuse — each canvas recomputes all KV. 3. MoE efficiency: 128 fine-grained experts + 1 shared expert, top-8 routing; 26B total / 3.8B active parameters. Memory footprint: ~52GB BF16 (H100 / RTX PRO 6000), ~28GB INT8 (L40S), ~18GB NVFP4 (RTX 4090). 4. Unified state diffusion (not masked diffusion like LLaDA): all tokens start as pure noise and iteratively converge to valid text — closer to image diffusion than mask-reveal schemes.
Performance: A Brutal Speed-vs-Quality Trade-off
Speed
| Configuration | Tokens/s | vs. baseline | |---|---|---| | H100 FP8, batch=1 | 1,100+ | 4x Gemma 4 AR | | H100 FP8, batch=4 | ~2,800 | 15x Gemma 4 AR | | RTX 5090, batch=1 | 700+ | — | | L40S INT8, T=10, batch=1 | ~480 | ~7x Gemma 4 AR | | L40S INT8, T=10, batch=4 | ~1,400 | — |
Advantage is strongest at small batch / low latency; at batch > 32, AR models with KV-cache reuse overtake it.
Quality (official benchmarks, vs Gemma 2.0 Flash-Lite)
| Benchmark | DiffusionGemma | Gemma 2.0 Flash-Lite | Gap | |---|---|---|---| | LiveCodeBench | 30.9% | 28.5% | +2.4% ✅ | | BigCodeBench | 45.4% | 45.8% | -0.4% | | HumanEval | 89.6% | 90.2% | -0.6% | | GPQA Diamond | 40.4% | 56.5% | -16.1% ⚠️ | | AIME 2025 | 23.3% | 20.0% | +3.3% ✅ | | BIG-Bench Hard | 15.0% | 21.0% | -6.0% ⚠️ | | Global MMLU | 69.1% | 79.0% | -9.9% ⚠️ |
Code tasks are on par or slightly ahead; scientific reasoning and multilingual knowledge lag notably. Google itself concedes DiffusionGemma scores below standard Gemma 4 on public benchmarks. Likely causes: accumulated denoising error, instability of long-range dependencies without chain propagation, canvas-boundary stitching artifacts, and a training objective (denoising) misaligned with final-text-quality evaluation.
Deployment Notes (from a hands-on video test)
Why It Matters
1. Paradigm shift: inference moves from memory-bandwidth-bound to compute-bound, with implications for GPU design, quantization payoffs, and cost models (AR cost ∝ output length; diffusion cost ∝ length × denoising steps). 2. Structural advantage in fill-in-the-middle (FIM): bidirectional attention uses prefix and suffix simultaneously; Google's Sudoku demo went from 0% to 80% accuracy exploiting constraint propagation in all directions. 3. Open ecosystem: Apache 2.0 weights on Hugging Face enable fine-tuning, hybrid AR-prefix + diffusion-generation research, and llama.cpp support (dedicated branch). 4. Local AI speed: 700+ tok/s on RTX 5090 means consumer-GPU local LLMs reach perceptually instant generation for the first time.
Limitations and Open Questions
Diffusion Language Model Timeline
| Date | Model | Scale | Significance | |---|---|---|---| | 2024 | SEDD (Stanford) | academic | discrete diffusion foundations | | 2025 | LLaDA | 8B | MMLU gap to AR narrowed to ~5pt | | 2025 | Mercury (Inception Labs) | commercial | 737-1,109 tok/s on H100, code-focused | | 2025 | Gemini Diffusion | closed | Google internal validation | | 2026.6 | DiffusionGemma | 26B MoE | first large-scale open-source DLM |