Background: Induction Heads in Autoregressive Models
If you have followed Transformer interpretability research, you have likely encountered the induction head — the most thoroughly studied circuit in autoregressive (AR) language models. Its mechanism is simple: when the model sees a pattern A B in context and later encounters A, it predicts B as the next token. Two attention head types cooperate: a previous-token head writes "what was the previous token" into the residual stream, and an induction head uses that information to match the earlier pattern and copy the token that followed it.
This circuit is considered one of the core mechanisms behind in-context learning. Olsson et al. (2022) described it as the smallest unit by which Transformers "learn to learn."
But all of this research focused on AR models. For the rising class of Diffusion Language Models (DLMs), almost nothing was known.
Diffusion Language Models: A Different Generation Paradigm
DLMs do not predict the next token. Instead, they iteratively denoise from noise, generating text the way diffusion models generate images. Given partially masked text, a DLM predicts the masked content and iterates until all positions are filled.
Attractive properties include bidirectional attention (every position sees both left and right context), non-left-to-right generation (parallel decoding), and a controllable, iterative process. Models like Mercury and DiffuGemma have shown commercial potential. But what happens inside DLMs? Do they have "induction heads," and are they the same as AR models'?
In July 2026, Andy Catruna and Emilian Radoi of the University Politehnica of Bucharest published the first systematic dissection of DLM internal circuits.
Finding: DLMs Learn "Bidirectional Induction Heads"
The researchers trained architecture-matched AR models and DLMs (simplified attention-only Transformers without FFNs) and compared their circuits. The finding in one sentence: DLMs learn induction head circuits isomorphic to AR models — but bidirectional.
- AR induction head: the previous-token head writes "the token to the left" into the residual stream → the induction head matches context → copies the following token. Looks left only.
- DLM induction head: a previous-token head plus a next-token head write both the left and right neighbor tokens into the residual stream → the induction head matches context using cues from both directions → copies the answer token. Looks both ways.
- Condition 1: the DLM sees only left-side context (mimicking an AR model's view)
- Condition 2: the DLM sees context on both sides (its normal operating mode)
- A linear probe can recover the global mask ratio from a single residual-stream position with high accuracy
- Patching (intervening) along this "mask-ratio direction" causally changes prediction entropy
- The simplified models (attention-only, no FFN) may differ from real large-scale DLMs
- Only the induction circuit was analyzed; DLMs may have other unique circuits
- The implicit timestep finding was made on small models; whether it holds for commercial-scale DLMs like Mercury is unknown
The circuit is direction-symmetric: the same circuit works whether the answer lies to the left or right of the masked position. No separate "look forward" and "look backward" mechanisms are learned.
Key Experiment: Mask One Side, Measure the Drop
To verify that bidirectionality is the source of the advantage:
Result: under Condition 1, DLM induction ability shows no significant difference from AR models. Under Condition 2, the DLM is significantly stronger.
This directly shows the DLM's advantage is not a stronger unidirectional mechanism, but its ability to exploit bidirectional context — same circuit, more information, better performance.
Striking Bonus Finding: Emergent "Implicit Timesteps"
In image diffusion models, the timestep (how far denoising has progressed) is usually injected via explicit embeddings. The researchers found that DLMs learn to infer the timestep from the global mask ratio without any explicit timestep embedding:
In other words, DLMs spontaneously develop an internal "timestep estimator" that reads how much of the sequence is masked to gauge denoising progress and adjust predictions accordingly. This capability emerged during training rather than being designed in.
This contrasts sharply with AR models, which never need a "timestep" concept since they only see already-generated left-side tokens. DLMs face a partially filled sequence and must know "how far along is denoising" to make sensible predictions — and they learn to extract that information themselves.
Why It Matters
1. Mechanistic interpretability extends from AR to DLMs
For five years, Transformer interpretability research has been dominated by AR models. This paper opens the door for DLMs: they are not learning something entirely different — they extend AR circuits with directionality. The previous-token head becomes a previous-token + next-token pair; unidirectional matching becomes bidirectional. Same circuit template, one extra dimension.
2. Bidirectional context is the root of the DLM advantage
The paper gives direct evidence that, at least for induction tasks, bidirectional context access is the core source of DLMs' advantage. Restrict the DLM to left-only context and it degrades to AR level; open up the right side and it becomes stronger.
3. Implicit timesteps are emergent computation
DLMs' ability to estimate timesteps means models can extract global state information from input statistics even when it is not explicitly provided — a minimal form of something like metacognition.
Open Questions
The paper candidly notes limitations:
---
Analysis tooling: TransformerLens