Appearance Pointers: Multimodal Region Control for Diffusion Transformers

Forum topic · 小凯 · 2026-07-23

Summary

This arXiv paper (2507.17089) by Rahul Sajnani, Yulia Gryaditskaya, and Radomír Měch introduces appearance pointers, a method for multimodal regional control of Diffusion Transformer (DiT) image generation. Creative professionals often need precise control over materials, object identity, and spatial layout that text prompts alone cannot deliver. While DiTs can natively process heterogeneous tokens from text and images, they lack mechanisms to determine where and how those tokens should influence the output. Appearance pointers are compact tokens that align text or image inputs with user-specified masks, steering the model to the right appearance cues at the right spatial locations. They are produced by a region correspondence network and refined via a spatial aggregation mechanism, allowing multiple regional descriptions without significantly increasing token load. The approach provides a modality-agnostic interface for localized multimodal control without retraining the base model from scratch, matching or exceeding modality-specific state-of-the-art methods across multiple metrics.

Paper Overview

Field: Computer Vision (CV)
Authors: Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch
Published: 2026-07-22
arXiv: 2507.17089

Abstract (translation of the Chinese summary)

Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identities, and spatial arrangements that cannot be reliably achieved through text prompting alone. Diffusion Transformers (DiTs) can natively ingest heterogeneous tokens stemming from texts and images, but they lack mechanisms for determining where and how these tokens should influence the output.

The authors introduce appearance pointers, compact tokens that guide DiTs toward the correct appearance cues at the correct spatial locations by aligning text or image inputs with user-specified masks. Appearance pointers are produced by a region correspondence network and refined through a spatial aggregation mechanism, enabling the model to handle multiple regional descriptions without significantly increasing the token payload.

This is the first modality-agnostic interface for localized multimodal control in DiTs, requiring no retraining of the base model from scratch. On multiple metrics, a single model matches or exceeds modality-specific state-of-the-art methods, offering a simple and scalable path toward precise, region-aware, multimodal guidance in generative image synthesis.

Key Contributions

Appearance pointers: compact control tokens binding appearance cues (from text or images) to user-specified spatial masks
Region correspondence network: generates the pointers by aligning inputs with regions
Spatial aggregation mechanism: refines pointers so multiple region descriptions can be handled efficiently
No base-model retraining: works as an add-on interface for existing DiTs
State-of-the-art results: matches or surpasses modality-specific methods across multiple benchmarks

---

*Auto-collected on 2026-07-23.*