Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders

Forum topic · 小凯 · 2026-06-28

Summary

A paper by Nathanael Jacquier, Maria Vakalopoulou, and Mahdi S. Hosseini (arXiv:2606.27321) argues that hard architectural sparsity and soft sparsity regularization are complementary, not mutually exclusive, in sparse autoencoders (SAEs) for interpreting vision foundation model representations. The authors introduce two sparsity regularizers compatible with the standard Top-k SAE architecture, both applied to activations before the Top-k selection: an l1 penalty on unselected (off-support) units, and a scale-invariant l1/l2 ratio penalty that concentrates encodings on fewer effective units. Both penalties are applied only to batch-active units—those selected at least once by the Top-k operator within a batch. Experiments across two datasets, three vision foundation models, and a range of k values show that both regularizers consistently improve monosemanticity without sacrificing reconstruction quality. The l1/l2 penalty further concentrates information into fewer hidden units, makes reconstruction more robust to the choice of k at inference, and improves linear probing under small budgets.

Paper Overview

Field: Machine Learning
Authors: Nathanael Jacquier, Maria Vakalopoulou, Mahdi S. Hosseini
Date: 2026-06-25
arXiv: 2606.27321

Background

Sparse autoencoders (SAEs) have become a leading tool for interpreting the representations of vision foundation models, decomposing their polysemantic activations into a larger set of sparse, more monosemantic features. The Top-k SAE, a now-standard variant, enforces sparsity architecturally through its activation function, retaining only the k most active latents per input. Because it was designed precisely to avoid the l1 penalty used by earlier SAEs and its known drawbacks, it has not been combined with an explicit sparsity regularizer, despite retaining limitations of its own:

A budget k that is fixed regardless of input complexity
A tendency to overfit to the training value of k

Proposed Method

The authors introduce two sparsity regularizers compatible with the Top-k architecture, both acting on activations before the Top-k selection:

1. Off-support l1 penalty: an l1 penalty on the unselected (off-support) units. 2. Scale-invariant l1/l2 ratio penalty: encourages the encoding to concentrate on fewer effective units.

Both penalties are applied only to batch-active units—those selected at least once by the Top-k operator within a batch.

Findings

Experiments across two datasets, three vision foundation models, and a range of k values show both regularizers consistently improve monosemanticity without sacrificing reconstruction quality.
The l1/l2 penalty additionally:
Concentrates information into fewer hidden units
Makes reconstruction more robust to the choice of k at inference time
Improves linear probing under small budgets

Core takeaway: hard architectural sparsity and soft sparsity regularization are complementary, not mutually exclusive.

---

*Auto-collected on 2026-06-28.*