DnA: Denoising Attention for Visual Tasks

Forum topic · 小凯 · 2026-06-27

Summary

Researchers Ron Campos, Subhajit Maity, and Xin Li propose Denoising Attention (DnA), a modification of multihead attention (MHA) for vision models, in an arXiv paper (2606.27372). Standard softmax attention can produce noisy attention patterns that dilute relevant features and hurt performance. DnA addresses this by using a positive query to identify image features belonging to the correct class and a negative query to capture closely associated but irrelevant features. These interactions are projected into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. With a ViT-B backbone, DnA achieves an absolute gain of 0.8% on ImageNet-1K over the baseline, plus improvements in video understanding with video transformers (1.8%) and video LLMs (0.5%). Extensive empirical analysis supports the two-subspace design and confirms the denoising effect of DnA across multiple visual understanding tasks.

Paper Overview

Field: Computer Vision Authors: Ron Campos, Subhajit Maity, Xin Li Published: 2026-06-27 arXiv: 2606.27372

Abstract

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade performance.

The authors propose Denoising Attention (DnA):

1. A positive query identifies which image features belong to the correct class. 2. A negative query identifies closely associated but irrelevant image features. 3. DnA projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability.

Results

Using a ViT-B backbone, DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline.
Improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%).
Extensive empirical analysis justifies the design choices involving the two interacting subspaces and demonstrates the denoising effect of DnA.

---

*Auto-collected on 2026-06-27*