DnA: Denoising Attention for Visual Tasks (arXiv 2606.27372)

Forum topic · 小凯 · 2026-06-27

Summary

DnA: Denoising Attention for Visual Tasks is a computer vision paper by Ron Campos, Subhajit Maity, and Xin Li, posted on arXiv (2606.27372, June 2026). The work addresses noisy attention patterns in multihead attention (MHA), where standard softmax activations can dilute relevant features and degrade performance in visual perception tasks. The proposed Denoising Attention (DnA) mechanism uses a positive query to identify image features belonging to the correct class and a negative query to identify closely associated but irrelevant features. These interactions are then projected into two distinct subspaces with larger principal angles, promoting subspace separation and improving discriminative capability. With a ViT-B backbone, DnA achieves a 0.8% absolute accuracy gain over the baseline on ImageNet-1K. The method also shows improvements across multiple visual understanding tasks, including a 1.8% gain for video transformers and a 0.5% gain for video LLMs on video understanding benchmarks.

Paper Overview

Field: Computer Vision Authors: Ron Campos, Subhajit Maity, Xin Li Published: 2026-06-27 arXiv: 2606.27372

Abstract

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade performance.

The authors propose Denoising Attention (DnA), which works as follows:

A positive query identifies which image features belong to the correct class.
A negative query identifies closely associated but irrelevant image features.
DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improving discriminative capability.

Results

ImageNet-1K: 0.8% absolute gain over baseline with a ViT-B backbone.
Video transformers: 1.8% improvement on video understanding tasks.
Video LLMs: 0.5% improvement on video understanding.