English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders

Forum topic · 小凯 · 2026-08-13

Summary

This paper by Nikolai Bolik, Lennart Stöpler, and Artur Andrzejak (arXiv:2608.11197) examines whether sparse autoencoder (SAE) latent sets behave as compositional 'bags of features' in large language models. The authors revisit Shani et al.'s (2026) analysis of human category structure, replacing cosine similarity over dense representations with overlap between active SAE latent sets, a more interpretable similarity measure. They first validate the measure: SAE latent sets recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. However, when extending the human-concepts analysis to SAE set similarity, they find SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states; instead, they track the model's internal similarity structure. Studying active latent sets under controlled semantic modifications reveals a significant mismatch between human judgments of concept change and changes in SAE active sets. The authors interpret this as evidence that, outside idealized settings, SAE features do not compose via simple bag-of-features semantics.

Overview

  • Field: Machine Learning
  • Authors: Nikolai Bolik, Lennart Stöpler, Artur Andrzejak
  • Published: 2026-08-11
  • arXiv: 2608.11197

Abstract (translated)

Shani et al. (2026) show that LLM representations broadly recover human category boundaries, but fail to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations. The authors revisit this approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure.

They first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. Extending the human-concepts analysis to SAE set similarities, they find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states — instead, they track the model's internal similarity structure.

To probe this gap further, the authors study active latent sets under controlled semantic modifications, revealing a significant mismatch between human judgments of concept change and changes in SAE active sets. They interpret this as evidence that, outside idealized settings, SAE features do not compose via simple bag-of-features semantics.

Original Abstract

Shani et al. (2026) show that LLM representations broadly recover human category boundaries, while failing to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations. We revisit their approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure. We first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. Extending the human-concepts analysis to SAE set similarities, we find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states...

--- *Auto-collected on 2026-08-13*

Tags

#sparse-autoencoders#interpretability#llm#machine-learning#concepts#categorization#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633402