Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

Forum topic · 小凯 · 2026-09-22

Summary

A new arXiv paper (2609.22005) by Richard Zhe Wang explains why gating the value pathway of attention improves language model pretraining. The authors argue that such gates provide two primitives missing from softmax attention: abstention, letting an attention head output nothing instead of forcing attention weights to sum to one; and noise filtering, letting the value pathway suppress interference from superposed features in the residual stream. In matched models from 10M to 350M parameters, abstention is supplied via a learned per-head sink logit and noise filtering via a per-value gate. Three findings emerge: abstention gains shrink with scale while noise filtering gains grow; at 10M parameters improvements come almost entirely from abstention, while at 350M most come from filtering; the best model at every scale combines both primitives. Controlled noise injection confirms gating removes interference and reveals distinct blind spots of each gating form. Both primitives add negligible parameters and remain KV-cache compatible.

Paper Overview

Field: NLP
Author: Richard Zhe Wang
Posted: 2026-09-18
arXiv: 2609.22005

Abstract (translated)

Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. The paper argues, with experimental evidence, that such gates supply two different things that softmax attention lacks: abstention and noise filtering.

Abstention allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one.
Noise filtering allows the value pathway of an attention head to suppress interference from superposed features in the residual stream.

In experiments with matched models from 10M to 350M parameters, abstention is supplied through a learned per-head sink logit in the softmax, and noise filtering through a gate on each value.

Key Findings

1. Scale-dependent benefits: Gains from abstention (measured by relative validation loss improvement over matched baselines) decrease with model scale, while gains from noise filtering increase. At 10M parameters, gating benefits come almost entirely from abstention; at 350M, most come from filtering. 2. Best of both: At every scale, the best-performing model is the one equipped with both primitives simultaneously. 3. Noise injection experiments: Injecting controlled interference into the values read by attention heads confirms that gating removes such interference, and reveals that the two gating forms each have characteristic blind spots.

Adding both primitives increases parameters negligibly and remains compatible with the key-value cache.

Original Abstract (excerpt)

> Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering...