dQwen3.5: Turning Hybrid-Attention Qwen3.5 Models into Diffusion Language Models

Forum topic · 小凯 · 2026-09-20

Summary

A new arXiv paper (2609.20751) by researchers including Anton Xue, Litu Rout, Aditya Akella, Adam Klivans, Sujay Sanghavi, and Sanjay Shakkottai explores adapting pretrained hybrid-attention language models into diffusion language models (DLMs). While most AR-to-diffusion conversions start from full-attention transformers, modern autoregressive models increasingly interleave attention with RNN layers, which are structurally causal and hard to bidirectionalize. The authors adapt Qwen3.5 at 0.8B, 2B, 4B, and 9B scales to create the dQwen3.5 family. Their key finding: hybrid backbones are efficient adaptation starting points, reaching a given training loss with roughly half the tokens compared to a full-attention control. Across all scales, dQwen3.5 exhibits any-order decoding behavior similar to full-attention DLMs and performs strongly under parallel decoding, demonstrating that hybrid architectures are viable for diffusion-based language modeling.

Paper Overview

Field: NLP Authors: Anton Xue, Litu Rout, Aditya Akella, Adam Klivans, Sujay Sanghavi, Sanjay Shakkottai Published: 2026-09-17 arXiv: 2609.20751

Summary

Adapting a pretrained autoregressive (AR) model is a cost-efficient route to building a diffusion language model (DLM). Nearly all such adaptations start from a full-attention transformer, but AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize.

Despite this mismatch, the authors investigate whether such backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, yielding the dQwen3.5 family.

Key Findings

Hybrid backbones are efficient starting points: compared against a full-attention control, the hybrid backbone reaches a given training loss in about half the tokens.
Comparable decoding behavior: across all scales, dQwen3.5 exhibits any-order decoding behavior similar to full-attention DLMs.
Strong parallel decoding: dQwen3.5 performs strongly when decoding in parallel.

Significance

This work demonstrates that the mismatch between causal RNN layers and diffusion modeling is not a blocker — hybrid-attention AR models can serve as efficient foundations for diffusion language models, potentially reducing training costs for future DLMs.

Links: arXiv:2609.20751

*Auto-collected on 2026-09-20.*

dQwen3.5: Turning Hybrid-Attention Qwen3.5 Models into Diffusion Language Models

Summary

Paper Overview

Summary

Key Findings

Significance

Tags