Circuit Anatomy of Diffusion Language Models: How Bidirectional Induction Heads Work

The induction head, studied to death — in AR models

If you have followed Transformer interpretability research, you have heard of the induction head.

It is one of the best-understood circuits in autoregressive (AR) language models. The mechanism is simple: the model sees a pattern A B in context, and on encountering A again, predicts B. It is implemented by two attention heads cooperating — a previous-token head that writes "what the previous token was" into the residual stream, and an induction head that uses this to match the pattern in context and copy the following token.

This circuit is considered a core mechanism behind in-context learning. Olsson et al. (2022) famously described it as a minimal building block of how Transformers "learn to learn."

But all of this research concerns AR models. For the rising class of diffusion language models (DLMs), almost nothing was known.

Diffusion language models: a different generation paradigm

DLMs do not predict the next token; they iteratively denoise text from noise. Given partially masked text, a DLM predicts the masked content and repeats until all positions are filled.

The paradigm offers bidirectional attention, non-left-to-right (parallel) decoding, and a controllable, correctable generation process. Models like Mercury and DiffuGemma have shown commercial potential. But what happens inside DLMs? What do their induction heads look like?

In July 2026, Andy Catruna and Emilian Radoi (Polytechnic University of Bucharest) published the first systematic circuit-level anatomy of DLMs.

Finding: DLMs learn bidirectional induction heads

The researchers trained architecture-matched AR models and DLMs (attention-only Transformers without FFNs) and compared their circuits.

DLMs learn an induction-head circuit isomorphic to the AR one — but bidirectional.

• AR induction head: previous-token head writes left-neighbor info → induction head matches context → copies the following token. Looks left only.
• DLM induction head: previous-token + next-token heads write both left- and right-neighbor info → induction head matches using both cues → copies the answer token. Looks both ways.

The circuit is directionally symmetric: the same machinery works whether the answer lies left or right of the masked position. No separate forward/backward mechanisms are learned.

Key experiment: mask one side

To verify that bidirectionality drives the advantage:

• Condition 1: DLM sees only left-side context (AR-like view)
• Condition 2: DLM sees both sides (normal operation)

Result: under Condition 1, DLM induction ability shows no significant difference from AR models. Under Condition 2, DLMs are significantly stronger.

This directly demonstrates that the DLM's advantage lies not in a stronger unidirectional mechanism, but in exploiting bidirectional context — same circuit, more information, better performance.

More surprising: DLMs learn an "implicit timestep"

Diffusion models rely on the timestep concept — how far denoising has progressed. Image diffusion models inject it via explicit embeddings.

But the researchers found that DLMs learn to infer the timestep from the global mask ratio, without any explicit timestep embedding:

• A linear probe can recover the global mask ratio from a single position in the residual stream with high accuracy
• Patching along this "mask-ratio direction" causally changes prediction entropy

So DLMs spontaneously form a "timestep estimator": by observing how much of the sequence is masked, they infer denoising progress and adjust prediction strategy. This capability emerged during training rather than being designed in.

AR models never need this — they only see already-generated left context. DLMs face a partially filled sequence and must know "how far along are we?" to predict sensibly — and they learn to extract this themselves.

Why this matters

1. Mechanistic interpretability extends beyond AR

For five years, Transformer interpretability has been dominated by AR models. DLMs — now commercializing — were a black box. This paper opens the door: DLMs are not learning something entirely different, but extending AR circuitry with directionality. Previous-token heads become previous-token + next-token heads; unidirectional matching becomes bidirectional. Same circuit template, one added dimension.

2. Bidirectional context is the root of DLM advantage

The source of DLMs' edge has been debated — bidirectional attention? iterative denoising? training objective? This paper gives direct evidence: at least for induction, bidirectional context access is the core source. Restrict the DLM to left context and it degrades to AR level; open the right side and it wins.

3. Implicit timestep is emergent computation

That DLMs learn to estimate the timestep means models can extract global state information from input statistics even when it is not explicitly provided. It resembles a minimal form of metacognition — inferring one's own processing state rather than being told.

Open questions

The authors honestly note limitations:

• Simplified models (attention-only, no FFN) may differ from real large-scale DLMs
• Only the induction circuit was analyzed; other unique DLM circuits may exist
• The implicit timestep finding comes from small models; generalization to commercial DLMs like Mercury is unknown

Still, as a first step in opening the black box, the paper is striking: different generation paradigms are not learning entirely different things — they extend the same circuit framework differently. AR chose efficiency (unidirectional, one pass); DLM chose flexibility (bidirectional, iterative). Circuit-level differences map precisely to this design-philosophy difference.

---

Paper: Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models

Analysis tooling: TransformerLens