How Transparent is DiffusionGemma? Measuring LLM Reasoning Transparency in Diffusion Language Models

Forum topic · 小凯 · 2026-06-20

Summary

This paper investigates the reasoning transparency of DiffusionGemma, a diffusion-based large language model that performs much of its computation in a continuous latent space. The authors decompose transparency into variable transparency (understanding intermediate snapshots of the model's computational state) and algorithmic transparency (reconstructing how the model derives outputs from those snapshots). Although DiffusionGemma initially appears to have poor variable transparency—with opaque serial depth (serial computation between interpretable states) 28.6x higher than autoregressive Gemma 4—the authors show that information flowing between denoising steps can be mapped through an interpretable token bottleneck without hurting downstream performance, reducing opaque serial depth to just 1.1x that of Gemma 4. Algorithmic transparency is harder for diffusion models because all token predictions can change at every denoising step, enabling distributed algorithms. Case studies reveal preliminary evidence of diffusion-specific phenomena including non-chronological reasoning, token and sequence smearing, and intermediate-context reasoning. Monitorability tests show DiffusionGemma and Gemma 4 are comparably monitorable.

How Transparent is DiffusionGemma?

arXiv: 2506.16807 Authors: Joshua Engels, Callum McDougall, Bilal Chughtai Published: 2025-06-20

Original Abstract

LLM reasoning transparency is a critical affordance for understanding model decisions, mitigating misuse and misalignment, and debugging surprising model behaviors. However, DiffusionGemma performs a larger fraction of its computation in a continuous latent space; does this make its reasoning less transparent? We study this question by decomposing transparency into two components: variable transparency, whether we understand intermediate snapshots of a model's computational state; and algorithmic transparency, whether we can use these snapshots to reconstruct the process by which the model arrived at its outputs. Naively, DiffusionGemma has poor variable transparency: its opaque serial depth, the amount of serial computation that occurs in between interpretable model states, seems at first...

Key Points

Motivation: Diffusion language models carry out a larger share of computation in a continuous latent space, raising the question of whether this makes their reasoning less transparent than autoregressive LLMs.
Framework: Transparency is decomposed into *variable transparency* (do we understand intermediate snapshots of the model's computational state?) and *algorithmic transparency* (can we reconstruct how the model arrives at outputs from these snapshots?).
Variable transparency finding: DiffusionGemma's opaque serial depth initially appears 28.6x higher than autoregressive Gemma 4, but information flowing between denoising steps can be mapped through an interpretable token bottleneck without degrading downstream performance. After this, opaque serial depth drops to only 1.1x that of Gemma 4.
Algorithmic transparency: Harder for diffusion models because all token predictions in the canvas can change at every denoising step, allowing the model to implement complex distributed algorithms during denoising.
Case studies: Preliminary evidence of diffusion-specific phenomena, including:
Non-chronological reasoning
Token and sequence smearing
Intermediate-context reasoning
Monitorability: Tests show DiffusionGemma and Gemma 4 have comparable monitorability.

*Source: zhichai.net forum post, originally from arXiv. Collected 2026-06-20.*