Learning to Read the Contextual Tokens in Diffusion Transformers

Forum topic · 小凯 · 2026-10-07

Summary

This paper introduces a framework for reading the contextual tokens inside multimodal diffusion Transformers (MM-DiT) via natural-language interrogation. MM-DiT models jointly process visual and textual representations throughout generation, repeatedly updating text tokens through multimodal attention to form dynamic contextual tokens whose function was previously unclear. The authors train a lightweight bottleneck network that maps intermediate contextual tokens into the input space of a frozen LLM, allowing the LLM to answer questions about the image being generated directly from these latent representations. The reader reveals that contextual tokens encode rich global scene representations: semantics—including attributes not specified in the prompt—become readable early in denoising, while finer details emerge later. Notably, information remains decodable even when the model receives an empty prompt, showing contextual tokens accumulate image-specific information from evolving visual representations. Generations with more readable contextual representations tend to score higher on human preference. Building on these findings, the authors propose Contextual Alignment, a training technique that explicitly reinforces visual-semantic information in contextual tokens, improving generation quality and distribution coverage.

Learning to Read the Contextual Tokens in Diffusion Transformers

Research area: Diffusion Authors: Omer Dahary, Etai Sella, Hadar Averbuch-Elor, Daniel Cohen-Or, Or Patashnik Published: 2026-10-05 arXiv: 2610.06844

Summary

Multimodal diffusion Transformers (MM-DiT) jointly process visual and textual representations throughout the generation process. These models repeatedly update the text tokens via multimodal attention, forming dynamic contextual tokens whose function has remained unclear. The authors introduce a framework for interrogating this contextual space through natural language. A lightweight bottleneck network is trained to map intermediate contextual tokens into the input space of a frozen large language model (LLM), enabling the LLM to answer questions about the image being generated directly from these latent representations.

Key Findings

Rich global scene representations: The reader reveals that contextual tokens encode global scene information. Semantics of the generated image—including attributes not specified in the prompt—become accessible surprisingly early in the denoising process, while increasingly fine-grained details become readable over time.
Information beyond the prompt: Notably, information remains decodable even when the MM-DiT receives an empty prompt, indicating that contextual tokens accumulate substantial image-specific information from the evolving visual representations themselves.
Correlation with quality: Generations whose contextual representations are more readable tend to achieve higher human preference scores.

Contextual Alignment

Building on these findings, the authors propose Contextual Alignment, a training technique that explicitly reinforces visual-semantic information within contextual tokens, improving both generation quality and distribution coverage.

Original Abstract

> We introduce a framework for reading contextual tokens in MM-DiTs through natural-language interrogation. Contextual tokens encode rich global scene representations accessible surprisingly early in denoising. We further propose Contextual Alignment, a training technique that improves generation quality by reinforcing visual-semantic information in contextual tokens.

*Auto-collected on 2026-10-07*

Learning to Read the Contextual Tokens in Diffusion Transformers

Summary

Learning to Read the Contextual Tokens in Diffusion Transformers

Summary

Key Findings

Contextual Alignment

Original Abstract

Tags