dQwen3.5: When Diffusion Language Models Meet Hybrid Architectures — Change Only a Quarter, Get Better Results

A Counterintuitive Finding

Imagine renovating an old house whose plumbing was designed strictly "left to right." You want an open layout where any room can be worked on simultaneously. The obvious approach is to tear down every wall — but someone tells you: just open up the partitions between the 25% load-bearing walls, keep the other 75% of pipes untouched, and the renovation goes faster and better.

That is essentially what dQwen3.5 does.

Background: The Awkward Position of Diffusion LMs

Two concepts first:

Autoregressive (AR) models: like GPT and Qwen — generate text strictly left to right, one token at a time.

Diffusion language models (DLMs): a newer paradigm. Starting from a fully masked "canvas," the model progressively unmasks positions, enabling parallel and arbitrary-order generation — like filling in a blank-filling exam, starting with the blanks you're most confident about.

DLMs promise faster parallel decoding and flexible ordering, but training a DLM from scratch is extremely expensive — roughly an order of magnitude more tokens than an AR model of equal quality. So researchers adapt existing AR models instead (AR-to-DLM adaptation). Prior work (DiffuLLaMA, Dream-7B, CoDA) all started from full-attention AR models.

The problem: modern AR models are no longer pure attention.

The Rise of Hybrid Architectures

Since 2025, state-of-the-art AR models — Qwen3.5, MiniMax-01, Jamba — use hybrid architectures: mostly RNN layers (specifically Gated DeltaNet) with only a minority of attention layers.

The reason is simple: RNN inference cost is O(1) per step regardless of context length, while attention is O(n). A 75% RNN + 25% attention model keeps attention's global view while gaining RNN efficiency. Qwen3.5 follows this recipe: the 0.8B model has 18 Gated DeltaNet layers + 6 attention layers; the 9B model has 24 + 8.

This creates a structural conflict for DLM adaptation.

The Structural Conflict

DLMs need arbitrary-order generation, which requires "seeing the future" — each position must access information from later positions. For attention layers this is easy: just remove the causal mask. But RNNs are inherently causal — they compress all previous positions into a hidden state passed position by position. You cannot bidirectionalize an RNN without changing its computational nature entirely (e.g., a BiRNN is a different model).

So dQwen3.5 makes a seemingly lazy decision:

> Bidirectionalize only the 25% attention layers; keep the 75% RNN layers causal.

The experimental results are surprising.

Why It Works

The key insight: natural language has a strong built-in left-to-right bias. Token prediction depends heavily on nearby left context, with selective contributions from distant context. Hybrid AR models encode this bias into the architecture — RNN layers accumulate left context sequentially, attention layers occasionally jump outward.

DLM decoding shows the same tendency: even with arbitrary-order decoding allowed, generation proceeds roughly left to right with occasional local disorder. So bidirectionalizing only the attention layers while keeping RNNs causal matches the nature of language itself: RNNs provide the left-to-right skeleton, attention layers provide the occasional "leaps."

A further technical detail: Token Shifting. dQwen3.5 uses the hidden state at position k to predict the token at position k+1, aligning the readout with AR pretraining — an alignment shown to significantly improve AR-to-DLM adaptation.

Results: Less Is More

Adaptation Speed

With comparable trunk parameter counts (1.37B vs 1.41B), dQwen3.5-2B needs only 45% of the tokens required by the full-attention control (dQwen3-1.7B) to reach the same training loss — a median 2.21x token saving.

Changing only 25% of the layers actually makes adaptation more than twice as fast.

Downstream Performance

After adapting on 50B tokens, dQwen3.5-2B across 7 benchmarks:

• Beats the full-attention control on 3/7 (HumanEval, MBPP, MBPP+ — all code tasks), often leading under parallel decoding

Compared with other DLMs (also 50B-token adaptation):

• vs CoDA (200B tokens): dQwen3.5-2B is better on 6/7 benchmarks
• vs Dream-7B (580B tokens): dQwen3.5-9B is better on 4/7 benchmarks

Note the token budgets: dQwen3.5 uses only 1/11 of Dream-7B's training compute yet wins on most benchmarks.

Scale Effects

An interesting finding: the larger the model, the smaller the returns from longer adaptation.

• 0.8B: the 100B checkpoint beats the 50B one on 5/7 benchmarks
• 9B: the 100B checkpoint beats the 50B one on only 1/7; on 6/7 the 50B checkpoint is better

For large models, 50B tokens suffice, and continued training degrades knowledge and math tasks — likely tied to the code-heavy data mix (50% code + 35% general + 15% math).

Decoding Behavior

dQwen3.5-9B's decoding closely resembles the from-scratch LLaDA-8B: broadly left-to-right with frequent local disorder. The hybrid architecture does not change the essential character of diffusion generation.

Deeper Implications

1. Minimal intervention: don't change everything you can — change only what you must. RNN causality is structural; attention causality is just a mask. Respecting the difference yields better results — the software-engineering principle of minimizing the change surface.

2. Rethinking proxy objectives: training loss and downstream performance are not the same thing. dQwen3.5-4B and 9B have nearly overlapping loss curves but clearly separated downstream scores. Training loss is a misleading proxy when comparing architectures.

3. Adaptability as an overlooked architectural dimension: beyond "performance per training token," architectures differ in how easily they convert across paradigms. Good architectures should be easy to retrofit — not just optimal within one paradigm.

Limitations and Open Questions

• Math remains weak: dQwen3.5 underperforms the full-attention control on math benchmarks, possibly due to RNN limitations on long reasoning chains
• Tokenizer differences: Qwen3 and Qwen3.5 tokenizers differ (4.18 vs 4.32 bytes/token); strict token-level comparisons need correction
• Base models only: post-training (SFT/RLHF) effects were not evaluated — the paper calls this a "separate problem"

Open Source

dQwen3.5 releases base models at four scales (0.8B/2B/4B/9B), plus full-attention controls, providing a unified reference for studying hybrid-architecture AR-to-DLM adaptation.

• Paper: https://arxiv.org/abs/2609.20751
• Qwen3.5 base models: https://github.com/QwenLM/Qwen3.5

One-line takeaway: dQwen3.5 shows that converting a hybrid architecture into a diffusion LM requires only bidirectionalizing the 25% attention layers while keeping the 75% RNN layers causal — adaptation is ~2.2x faster in tokens, and downstream performance beats controls trained with 4–11x more tokens. Less is more; respecting structural differences beats forcing uniformity.