Paper Overview
- Research Area: Machine Learning (Mechanistic Interpretability)
- Authors: Nikolai Bolik, Lennart Stöpler, Artur Andrzejak
- Release Date: 2026-08-11
- arXiv: 2608.11197
- Method: Replaces cosine similarity on dense embeddings with set overlap over active SAE latents, arguing set-level comparison is more interpretable than point-vector similarity.
- Validation on toy models: SAE latent sets successfully recover union-like compositional structure, and on natural text induce semantically coherent neighborhoods.
- Human-concepts analysis extended: When applied to SAE set similarities, active sets do not outperform dense embeddings or residual-stream states at recovering human category boundaries or within-category typicality; instead they mirror the model's own similarity structure.
- Semantic modification experiments: Under controlled semantic perturbations, changes in active latent sets diverge sharply from human judgments of conceptual change.
- Interpretation: Evidence that SAE features do not generally compose via a simple bag-of-features semantics; set-level "instability" is a real limitation of SAE-based interpretations beyond idealized settings.
- arXiv link: https://arxiv.org/abs/2608.11197
- Auto-collected 2026-08-13
Summary
Shani et al. (2026) show that LLM representations broadly recover human category boundaries, while failing to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations.
We revisit their approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure. We first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text.
Extending the human-concepts analysis to SAE set similarities, we find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states, but instead track model-internal similarity structure.
To probe this gap further, we study active latent sets under controlled semantic modifications, revealing a pronounced mismatch between human judgments of conceptual change and changes in SAE active sets. We interpret this as evidence that, outside idealized settings, SAE features do not compose via a simple bag-of-features semantics.
Original Abstract
> Shani et al. (2026) show that LLM representations broadly recover human category boundaries, while failing to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations. We revisit their approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure. We first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. Extending the human-concepts analysis to SAE set similarities, we find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states, but instead track model-internal similarity structure. To probe this gap further, we study active latent sets under controlled semantic modifications, revealing a pronounced mismatch between human judgments of conceptual change and changes in SAE active sets. We interpret this as evidence that, outside idealized settings, SAE features do not compose via a simple bag-of-features semantics.
---