DiffusionGemma Deep Dive: From Token-by-Token to Block-by-Block Generation

> Release: June 10, 2026, Google DeepMind, Apache 2.0 > Hugging Face: google/diffusiongemma-26B-A4B-it

One-Sentence Summary

DiffusionGemma is not a "faster autoregressive model" — it completely changes the generation paradigm, from GPT-style "token-by-token" to Stable-Diffusion-style "whole-block spray." Using a 256-token canvas + bidirectional attention + iterative denoising, it hits 1,100+ tok/s on H100 (4x a same-size AR model), at the cost of 5-20 benchmark points below Gemma 4. This is not a speed patch but a structural surgery that turns LLM inference from memory-bound into compute-bound.

The Core Problem: The Ceiling of Autoregressive Speed

All GPT/Claude/Gemma models are autoregressive (AR): each new token must wait for the previous one.

| Problem | Cause | Consequence | |---|---|---| | Memory-bound | KV cache read/write per token | GPU compute idle, bandwidth saturated | | No parallelism | Token N depends on N-1 | 256 tokens = 256 forward passes | | Redundant attention | Recomputed over history each step | Quadratic cost at long context |

Image diffusion models (Stable Diffusion) offer an alternative: start from noise and refine all pixels in parallel. DiffusionGemma asks: can text do the same, given strict sequential and grammatical dependencies?

Architecture: Encoder-Decoder + Block-Autoregressive Diffusion

Built on a Gemma 4 MoE backbone (26B total, 3.8B active) with a redesigned generation head:

• Encoder: standard causal-attention autoregressive encoder that processes the prompt and produces the KV cache.
• Decoder: a diffusion decoder with bidirectional (non-causal) attention over a 256-token canvas, refined over T denoising steps (10-48).

Key Design Decisions

1. Block-autoregressive: canvases are generated serially — after each 256-token canvas is denoised, the encoder absorbs it into the KV cache, and the next canvas continues from it. Parallelism within blocks, stability across blocks. 2. Bidirectional attention: every position sees all others in the canvas, including "future" tokens. Cost: no incremental KV cache reuse — each canvas recomputes all KVs. 3. MoE efficiency: 128 fine-grained experts + 1 shared expert, top-8 routing. VRAM needs: ~52GB BF16 (H100 / RTX PRO 6000), ~18GB NVFP4 (RTX 4090), ~28GB INT8 (L40S). 4. Unified state diffusion (not masked diffusion like LLaDA): all tokens start as pure noise and iteratively converge, closer to image diffusion intuition than progressive [MASK] reveal.

Performance: Speed vs. Quality

Speed

| Config | Tok/s | vs. Gemma 4 AR | |---|---|---| | H100 FP8, batch=1 | 1,100+ | 4x | | H100 FP8, batch=4 | ~2,800 | 15x | | RTX 5090, batch=1 | 700+ | — | | L40S INT8, T=10, batch=1 | ~480 | ~7x | | L40S INT8, T=10, batch=4 | ~1,400 | — |

The advantage is largest at small batch / low latency. At batch > 32, AR models with KV cache reuse overtake it.

Quality (Google official, vs. Gemma 2.0 Flash-Lite)

| Benchmark | DiffusionGemma | Flash-Lite | Gap | |---|---|---|---| | Code LiveCodeBench | 30.9% | 28.5% | +2.4% ✅ | | Code BigCodeBench | 45.4% | 45.8% | -0.4% | | Code HumanEval | 89.6% | 90.2% | -0.6% | | Science GPQA Diamond | 40.4% | 56.5% | -16.1% ⚠️ | | Math AIME 2025 | 23.3% | 20.0% | +3.3% ✅ | | Reasoning BIG-Bench Hard | 15.0% | 21.0% | -6.0% ⚠️ | | Multilingual Global MMLU | 69.1% | 79.0% | -9.9% ⚠️ |

Code tasks are at parity or slightly better; science, reasoning, and multilingual lag significantly. Google itself acknowledges DiffusionGemma underperforms standard Gemma 4 on all public benchmarks.

Why quality drops

1. Cumulative denoising error across T steps. 2. Bidirectional attention's "excess freedom" — long-range dependencies are less stable than AR's chain. 3. Seams at 256-token canvas boundaries. 4. Training objective (denoising) not fully aligned with final-text-quality evaluation.

Deployment Findings (video walkthrough, H100 via RunPod)

• Personal finance dashboard: working code, but more small errors to fix manually than AR models; whole code appears at once instead of typing line-by-line.
• Arcade-style game: core logic correct, edge cases and polish weaker; diffusion's "global planning" seems suited to architecture-level code.

Real-world throughput: ~800-1000 tok/s for short snippets, ~600-800 for long documents, ~400-600 for multi-turn dialogue (context re-encoded each turn).

Why DiffusionGemma Matters

1. Paradigm shift to compute-bound: future GPU design, quantization gains (compute savings rather than memory savings), and cost models (output length × denoising steps) all change. 2. Fill-in-the-middle advantage: bidirectional attention uses prefix and suffix simultaneously — Google's Sudoku demo jumped from 0% to 80% accuracy via constraint propagation from all directions. 3. Open ecosystem: Apache 2.0 weights on Hugging Face enable fine-tuning, hybrid AR-prefix + diffusion-generation research, and llama.cpp support (dedicated branch). 4. Local AI: 700+ tok/s on RTX 5090 means consumer GPUs reach perceptually instant generation for the first time.

Limitations and Open Questions

• Pareto frontier: 10 steps = fast/rough, 20 = balanced, 48 = near-AR quality with shrinking speed edge.
• Long-document coherence: canvas seams may create a "stitched" feel in papers or novels; longer canvases (512/1024) may be needed.
• UX: block-sprayed output lacks the "model is thinking" feedback of streaming tokens.
• No prefix caching: every turn re-encodes full context, hurting long conversations.
• Cutoff: training data ends January 2025; no audio input (unlike Gemma 4 12B).

References

• DiffusionGemma on Hugging Face: https://huggingface.co/google/diffusiongemma-26B-A4B-it
• Gemini Diffusion (closed): https://deepmind.google/models/gemini-diffusion/
• vLLM recipe: https://recipes.vllm.ai/Google/diffusiongemma-26B-A4B-it
• NVIDIA NIM docs: https://docs.api.nvidia.com/nim/reference/diffusiongemma-26b-a4b-it
• Awesome Agents analysis: https://awesomeagents.ai/models/diffusiongemma-26b/
• Spheron deploy guide: https://www.spheron.network/blog/deploy-diffusiongemma-gpu-cloud/
• ModelScope GGUF: https://modelscope.cn/models/unsloth/diffusiongemma-26B-A4B-it-GGUF
• Original video: https://www.youtube.com/watch?v=Dxn3BcSgsMY

Diffusion Language Model Timeline

| Time | Model | Scale | Significance | |---|---|---|---| | 2024 | SEDD (Stanford) | academic | Discrete diffusion theory | | 2025 | LLaDA | 8B | MMLU gap to AR narrowed to ~5pt | | 2025 | Mercury (Inception Labs) | commercial | 737-1109 tok/s on H100, code-focused | | 2025 | Gemini Diffusion | closed | Google internal validation | | 2026.6 | DiffusionGemma | 26B MoE | First large open-weight DLM |