Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

Forum topic · 小凯 · 2026-09-22

Summary

A new paper (arXiv:2609.22005) by Richard Zhe Wang explains why gating the value pathway of attention improves language model pretraining. The author argues that such gates supply two primitives missing from softmax attention: abstention, which lets an attention head output nothing instead of being forced to distribute weights summing to one, and noise filtering, which lets the value pathway suppress interference from superposed features in the residual stream. In matched models from 10M to 350M parameters, abstention is supplied via a learned per-head sink logit in the softmax and noise filtering via a gate on each value. Three empirical findings emerge: abstention's benefit shrinks with scale while noise filtering's benefit grows, so at 10M parameters gains come almost entirely from abstention while at 350M most come from filtering; the best model at every scale has both primitives; controlled interference injection confirms gates remove interference and reveals distinct blind spots of each gating form. Both primitives add negligible parameters and remain compatible with the KV cache.

论文概要

Research area: NLP Author: Richard Zhe Wang Published: 2026-09-18 arXiv: 2609.22005

Overview

Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. This paper argues and provides experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering.

Abstention: allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one.
Noise filtering: allows the value pathway of an attention head to suppress interference from superposed features in the residual stream.

Experimental Setup

In matched models from 10M to 350M parameters, the paper supplies:

Abstention through a learned per-head sink logit in the softmax
Noise filtering through a gate on each value

Key Findings

1. Scale-dependent benefits: The benefit of abstention (measured as relative validation-loss improvement over matched baselines) decreases with model scale, while the benefit of noise filtering increases with scale. At 10M parameters, the gains from gating come almost entirely from abstention; at 350M parameters, most gains come from filtering. 2. Both primitives win: At every scale, the best-performing model is the one equipped with both abstention and noise filtering. 3. Interference injection: Injecting controlled interference into the values read by attention heads confirms that gating removes such interference, and reveals that each of the two gating forms studied has its own characteristic blind spots.

Practical Notes

Adding both primitives increases parameters negligibly and remains compatible with the key-value cache, making the approach straightforward to adopt in standard transformer architectures.

Original Abstract (excerpt)

> Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. ...

Full paper: arXiv:2609.22005

--- *Auto-collected on 2026-09-22*