DnA: Denoising Attention for Visual Tasks (arXiv 2606.27372)

Forum topic · 小凯 · 2026-06-27

Summary

Researchers Ron Campos, Subhajit Maity, and Xin Li propose Denoising Attention (DnA), an alternative to the standard softmax activation in multihead attention for visual perception tasks. Standard softmax can produce noisy attention patterns that dilute relevant features and degrade performance. DnA uses a positive query to identify image features belonging to the correct class and a negative query to identify closely associated but irrelevant features, then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. With a ViT-B backbone, DnA achieves a 0.8% absolute gain on ImageNet-1K over the baseline, plus improvements in video understanding tasks (1.8% with video transformers, 0.5% with video LLMs). The paper is available at arXiv:2606.27372.

Paper Overview

Field: Computer Vision Authors: Ron Campos, Subhajit Maity, Xin Li Published: 2026-06-27 arXiv: 2606.27372

Original Abstract

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Extensive empirical analysis justifies the design choices involving the two interacting subspaces and the denoising effect of DnA.

Key Contributions

Replaces the standard softmax attention with a denoising mechanism using paired positive and negative queries
Projects positive/negative query-feature interactions into two distinct subspaces with larger principal angles for better discriminability
+0.8% absolute top-line gain on ImageNet-1K with a ViT-B backbone
+1.8% on video understanding with video transformers and +0.5% with video LLMs

--- *Auto-collected on 2026-06-27*