From SRA to Self-Flow: Data Augmentation or Self-Supervision?

Forum topic · 小凯 · 2026-07-04

Summary

This arXiv paper (2507.00477) investigates why Self-Flow outperforms SRA in self-representation alignment for diffusion transformers. Self-Flow attributes its gains to dual-time scheduling, where cleaner tokens help infer noisier ones via cross-noise-level attention. The authors challenge this explanation, proposing instead that the improvement stems from data augmentation along the noise dimension. To disentangle the two factors, they introduce Attention Separation, which keeps the same dual-timestep input but blocks attention between tokens assigned to different noise levels. Surprisingly, removing this interaction does not hurt performance and may even improve it, indicating the gains mainly come from data augmentation. They further show that Attention Separation itself provides augmentation by splitting a single image into multiple effective training parts. Combining self-representation alignment with dual-timestep and attention-separation augmentation, they demonstrate the design's effectiveness on ImageNet. Authors: Dengyang Jiang, Mengmeng Wang, Harry Yang.

Paper Overview

Field: Computer Vision Authors: Dengyang Jiang, Mengmeng Wang, Harry Yang Published: 2026-07-04 arXiv: 2507.00477

Abstract

Representation alignment has become an effective way to accelerate diffusion transformer training and improve generation quality. Recent self-alignment methods, such as SRA and Self-Flow, further remove the dependency on external pretrained encoders by constructing alignment within the diffusion model itself. However, the mechanism behind the improvement from SRA to Self-Flow, dual-time scheduling, remains under-examined: Self-Flow attributes its gain to interactions between tokens at different noise levels, where cleaner tokens help infer noisier ones.

In this work, the authors revisit this explanation and ask whether the gain instead comes from data augmentation along the noise dimension. To disentangle these factors, they introduce Attention Separation, which preserves the same dual-timestep input as Self-Flow while blocking attention between tokens assigned to different noise levels.

Key Findings

Surprisingly, removing the cross-noise-level interaction does not degrade performance and may even improve it, suggesting the improvement from SRA to Self-Flow mainly comes from data augmentation rather than cleaner-to-noisier token inference.
Attention Separation itself provides augmentation by splitting a single image into multiple effective training parts, thereby expanding training data.
Building on these observations, the authors combine self-representation alignment with dual-timestep and attention-separation augmentation, and demonstrate the effectiveness of this design on ImageNet.

---

*Auto-collected on 2026-07-04*