Sumi Deep Dive: A 7B Uniform Diffusion Language Model Pretrained from Scratch

> Paper: Sumi: Open Uniform Diffusion Language Model from Scratch > Authors: Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, Jun Suzuki (Tohoku University) > Link: https://arxiv.org/abs/2606.19005

Key points

• Sumi is the first fully open, from-scratch pretrained uniform diffusion LLM: 7B parameters, 1.5T training tokens, bidirectional Transformer, LLaMA-style architecture trained with Megatron-LM.
• It matches autoregressive (AR) baselines trained on the same token budget on standard benchmarks (HellaSwag, PIQA, ARC, HumanEval), but is weaker on commonsense reasoning — attributed to an education-heavy data mixture, not the diffusion paradigm itself.
• Everything is released: model weights, intermediate checkpoints, full training recipe, and public-corpus data mixture ratios.
• Exploratory findings: 2048 is the generation "canvas length" sweet spot; confidence-based sampling produces a self-organized generation order; an explicit self-correction budget did not improve quality (unlike GIDD's hybrid-noise results).
• Based on the GIDD framework (Generalized Interpolating Discrete Diffusion) with SNR reparameterization: λ = log(SNR) = log(α/β).

Why diffusion language models matter

AR models (GPT, Llama, Qwen) generate left-to-right and cannot revise already-emitted tokens; errors cascade. Generation is inherently serial (N tokens require N forward passes). Discrete diffusion research has largely stayed at small scale or fine-tuned pretrained AR models — until Sumi, no true from-scratch, large-scale uniform diffusion LM existed.

Three paradigms compared:

| Paradigm | Generation | Noise | Key trait | |:---|:---|:---|:---| | AR | Serial, left-to-right | None | Causal, no revision | | Masked Diffusion | Parallel denoising | [MASK] tokens | Revisable-ish, but noise is one state | | Uniform Diffusion | Parallel denoising | Uniform vocab sampling | Finer-grained, truly bidirectional and revisable |

Sumi uses pure uniform noise (m_t = 1/V) within GIDD's mixed-kernel formulation: q_t(z|x) = α(t)·δ(z,x) + β(t)·m_t(z).

Architecture and training

| Config | Value | |:---|:---| | Parameters | 7B | | Architecture | Bidirectional Transformer (32 layers, d=4096, 32 heads) | | Context | 2048 at generation time | | Tokenizer | GPT-2 BPE | | Training | Megatron-LM, two stages |

Training: ~1.3T tokens of education-weighted web/books/academic/wiki text (pretraining), then ~0.2T tokens of mid-training skewed toward code, math, and reasoning.

Performance: comparable to AR, but different distribution

• On standard benchmarks, Sumi is on par with AR models trained with the same token budget — a key proof that from-scratch diffusion scaling works at 7B/1.5T.
• On commonsense benchmarks (CommonsenseQA, Social IQa), Sumi trails AR, blamed on the education-skewed data mix rather than the paradigm.
• Diffusion offers parallel generation (latency tied to denoising steps, not sequence length) and token-level revisability that AR lacks.

Exploratory findings on generation behavior

1. Canvas length sweet spot: quality degrades below 1024 (space contention) and above 4096 (attention dilution); 2048 is optimal — unlike AR, there is no EOS-based termination. 2. Confidence sampling induces self-organized order: updating only the lowest-confidence tokens each step makes the model spontaneously fix structural "skeleton" tokens (punctuation, function words) before content words — dynamics unique to uniform diffusion. 3. Explicit self-correction budget fails: extra denoising steps reserved for correction did not improve output. Pure uniform noise may lack the self-correction behavior GIDD observed with hybrid masked+uniform noise, suggesting correction ability may need hybrid-noise training.

Scaling properties (from GIDD)

GIDD's scaling laws: masked diffusion α_M=0.566/α_D=0.434; balanced hybrid 0.534/0.466; uniform 0.589/0.411 — uniform diffusion is the most parameter-hungry. At 7B/1.5T, Sumi may be below its optimal parameter/data ratio; more parameters at fixed tokens could help.

Significance and limitations

Significance: validates diffusion as an independent path to large-model performance (not an AR derivative); provides an unusually clean, fully open research baseline; surfaces new generation-behavior findings.

Limitations: 2048 generation canvas limits long documents; commonsense gap from data mix; self-correction unrealized; GPT-2 BPE tokenizer is dated.

Future directions

• Hybrid noise schedules (possibly dynamic over training) to induce self-correction
• Adaptive canvas length per task
• Learnable confidence-sampling policies instead of hand-tuned ones
• Hybrid AR + diffusion architectures (e.g., BD³-LMs / Block Diffusion: https://arxiv.org/abs/2503.09573), where AR drafts structure and diffusion refines
• Inference optimization: fixed-step parallel generation maps well to GPU hardware

Conclusion

Sumi's value is not beating AR — it is mostly "comparable." Its real contribution is a clean, reproducible, fully open benchmark that lets the community study the diffusion paradigm's unique properties without training-difference confounds.

References:

• Sumi: https://arxiv.org/abs/2606.19005
• GIDD: https://arxiv.org/abs/2503.04482 | https://github.com/dvruette/gidd
• BD³-LMs: https://arxiv.org/abs/2503.09573
• Scaling Behavior of Discrete Diffusion: https://arxiv.org/abs/2512.10858
• Self-Correcting Discrete Diffusion (SCDD): https://arxiv.org/abs/2603.02230