dQwen3.5: Turning Hybrid AR Models into Diffusion LMs by Making Only 25% of Layers Bidirectional

A Counterintuitive Finding

Imagine renovating an old house. The entire plumbing and wiring were designed for a strict "left-to-right" order: the living room connects to the hallway, the hallway to the bedroom, water flows from the main pipe to the end. Now you want to convert it into an open layout where any room can be renovated simultaneously.

The most intuitive approach would be to tear down all the walls. But someone tells you: you only need to remove partitions between 25% of the load-bearing walls, keep the remaining 75% of pipes untouched, and the renovation finishes faster and better.

That's what dQwen3.5 does.

Background: The Awkward Position of Diffusion Language Models

Two concepts first.

Autoregressive models (AR): like GPT and Qwen, these LLMs generate text strictly from left to right, one token at a time. Like handwriting—you must write left to right, no skipping.

Diffusion language models (DLMs): a new paradigm. They start from a fully masked "canvas" and progressively unmask positions, generating in parallel and in arbitrary order. Like a fill-in-the-blank test—answer the blanks you're most confident about first.

DLMs are attractive: parallel generation means speed, arbitrary order means critical parts can be generated first. The problem is that training a DLM from scratch is extremely expensive—matching AR performance requires roughly an order of magnitude more tokens. So researchers ask: can we adapt existing AR models directly?

This is AR-to-DLM adaptation. Prior work (DiffuLLaMA, Dream-7B, CoDA, etc.) adapted full-attention AR models.

But here's the issue: modern AR models are increasingly not purely attention-based.

The Rise of Hybrid Architectures

Since 2025, the most advanced AR models—Qwen3.5, MiniMax-01, Jamba—use hybrid architectures: most layers are RNNs (specifically Gated DeltaNet), with only a minority of attention layers.

The reason is simple: RNN inference cost is O(1)—constant per-step computation regardless of context length. Attention costs O(n)—longer context means slower inference. So a 75% RNN + 25% attention hybrid retains attention's global view while gaining RNN efficiency.

Qwen3.5 is exactly this: the 0.8B model has 18 Gated DeltaNet layers + 6 attention layers; the 9B model has 24 Gated DeltaNet layers + 8 attention layers.

This creates a structural problem for DLM adaptation.

The Structural Conflict

DLMs' core capability is arbitrary-order generation. For that, the model needs to "see the future"—the current position must access information from later positions. For attention layers, this is easy: remove the causal mask, making them bidirectional.

But RNNs are inherently causal. Their core mechanism compresses all prior positions into a hidden state, passed position by position. You cannot "bidirectionalize" an RNN—except by turning it into a BiRNN, which fundamentally changes its computation and is no longer the same model.

So dQwen3.5 made a seemingly lazy decision:

> Bidirectionalize only the 25% attention layers; keep the remaining 75% RNN layers causal and unchanged.

This looks like cutting corners—how can you call it a "diffusion" model when only a small part changed? But the experimental results were surprising.

Why This Works

The key insight: natural language itself has a strong left-to-right bias.

This isn't a人为 convention—it's a property of language. Token prediction depends heavily on nearby left-side context, with distant context contributing selectively. Hybrid AR models encode this bias into the architecture: RNN layers accumulate left-side context position by position, while attention layers occasionally reach out for distant information.

DLMs at inference show a similar tendency: even with arbitrary-order decoding allowed, generation still proceeds roughly left to right, with occasional out-of-order adjacent positions.

So bidirectionalizing only the attention layers while keeping RNN causality matches the nature of language itself. RNNs provide a left-to-right skeleton; attention layers provide occasional "jumps."

One technical detail: Token Shifting. dQwen3.5 uses the hidden state at position k to predict the token at position k+1, aligning with the readout used during AR pretraining. This alignment has been shown to significantly improve AR-to-DLM adaptation.

Results: Less Is More

Adaptation Speed

With comparable trunk parameter counts (1.37B vs 1.41B), dQwen3.5-2B reaches the same training loss using only 45% of the tokens required by the full-attention control model (dQwen3-1.7B)—a median 2.21x token saving.

In other words: changing only 25% of the layers makes adaptation more than twice as fast.

Downstream Performance

After adapting with 50B tokens, dQwen3.5-2B across 7 benchmarks:

• Beats the full-attention control on 3/7 (HumanEval, MBPP, MBPP+—all code tasks)
• Frequently leads on code tasks under parallel decoding

Compared to other DLMs (also adapted with 50B tokens):

• vs CoDA (200B tokens): dQwen3.5-2B wins on 6/7 benchmarks
• vs Dream-7B (580B tokens): dQwen3.5-9B wins on 4/7 benchmarks

Note the token budgets: dQwen3.5 used only 1/11 of Dream-7B's training yet beats it on most benchmarks.

Scale Effects

An interesting finding: the larger the model, the smaller the gains from longer training.

• 0.8B: the 100B checkpoint beats the 50B one on 5/7 benchmarks
• 9B: the 100B checkpoint beats 50B on only 1/7 benchmarks; 50B is better on the other 6/7

For large models, 50B tokens suffices for adaptation; continued training causes regressions in knowledge and math tasks. This relates to the code-heavy training mix (50% code + 35% general + 15% math)—code tasks benefit from continued training, others don't necessarily.

Decoding Behavior

dQwen3.5-9B's decoding behavior closely resembles the from-scratch LLaDA-8B: overall left-to-right with frequent out-of-order adjacent positions. The hybrid architecture learns the same decoding patterns as pure-attention DLMs—architectural differences don't change the essential behavior of diffusion generation.

Deeper Implications

1. The "Minimal Intervention" Principle

dQwen3.5's success exemplifies minimal intervention: don't change everything you can—change only what you must. RNN causality is structural; forcing bidirectionality would break it. Attention causality is just a mask; removing the mask is enough. Respecting this difference yields better results.

This matches software engineering's "minimize the change surface" principle: the less you change, the less risk you introduce, and the faster adaptation goes.

2. Rethinking "Proxy Objectives"

Training loss and downstream performance are not the same thing. dQwen3.5-4B and 9B have nearly overlapping loss curves but clearly divergent downstream performance. Training loss, as a proxy objective, is misleading when comparing architectures.

This echoes the "proxy objective trap": optimizing a proxy variable isn't optimizing the real target. Here the proxy is training loss; the real target is downstream capability.

3. "Adaptability" Is an Overlooked Architecture Dimension

We usually evaluate architectures by "performance at equal training budget." But dQwen3.5 reveals another dimension: an architecture's adaptability to paradigm shifts. Hybrid architectures beat pure attention not only in inference efficiency but also in AR-to-DLM adaptation efficiency.

This suggests a broader rule: good architectures should be easy to transform—optimal not under one paradigm, but quick to adapt during paradigm transitions.

Limitations and Open Questions

• Math remains weak: dQwen3.5 underperforms the full-attention control on math benchmarks, possibly due to RNN limitations on long reasoning chains
• Tokenizer differences: dQwen3 and Qwen3.5 use different tokenizers (4.18 vs 4.32 bytes/token); strict token-level comparisons require correction
• Base models only: post-training (SFT/RLHF) effects weren't evaluated; the paper explicitly calls this a "separate problem"

Open Source

dQwen3.5 releases base models at four scales (0.8B/2B/4B/9B), plus full-attention control models, providing a unified reference for studying hybrid-architecture AR-to-DLM adaptation.

• Paper: https://arxiv.org/abs/2609.20751
• Qwen3.5 base models: https://github.com/QwenLM/Qwen3.5

One-sentence summary: dQwen3.5 shows that converting a hybrid architecture into a diffusion language model requires only bidirectionalizing the 25% attention layers while keeping the 75% RNN layers causal—adaptation is 2.2x faster, and downstream performance beats controls trained with 4–11x more tokens on most benchmarks. Less is more; respecting structural differences beats forcing uniformity.