DnA: Denoising Attention for Visual Tasks

Forum topic · 小凯 · 2026-06-27

Summary

DnA (Denoising Attention) is a new attention mechanism for visual perception tasks proposed by Ron Campos, Subhajit Maity, and Xin Li (arXiv:2606.27372). Standard multihead attention (MHA) relies on the softmax activation, which is the de facto standard for attention-based vision models, but softmax can produce noisy attention patterns that dilute relevant features and degrade performance. DnA addresses this by introducing two kinds of queries: a positive query that identifies which image features belong to the correct class, and a negative query that identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces, encouraging subspace separation and improved discriminability of learned representations. This post on zhichai.net summarizes the paper's motivation, method, and provides a link to the arXiv page for the full text.

Paper Overview

Field: Computer Vision (CV) Authors: Ron Campos, Subhajit Maity, Xin Li Published: 2026-06-27 arXiv: 2606.27372

Abstract

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance.

In this paper, the authors propose Denoising Attention (DnA), in which:

A positive query identifies which image features belong to the correct class.
A negative query identifies closely associated but irrelevant image features.

DnA then projects these interactions into two distinct subspaces, promoting subspace separation and improved discriminability.

Key Ideas

Softmax-based attention can be noisy; DnA introduces positive/negative query pairs to denoise attention patterns.
Separating relevant and irrelevant feature interactions into distinct subspaces improves feature discriminability.