English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

Beyond SAEs: Consistent Concept Discovery with Sparse Coding (ConSDL)

Forum topic · 小凯 · 2026-10-20

Summary

Sparse autoencoders (SAEs) interpret foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms, but learned dictionaries can vary across training runs, limiting reproducibility of SAE-based analyses. This paper proposes Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. The approach is motivated by synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with per-sample optimization, suggesting sparse coding is a more reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. On four vision and language settings, ConSDL delivers state-of-the-art cross-seed consistency, improving dictionary consistency by 17.6% and coefficient consistency by 35.9% relative to the average of six SAE baselines. Authored by Jinho Choi, Hyesu Lim, Jaegul Choo, and Steffen Schneider, the paper is available on arXiv as 2610.23801.

Paper Overview

  • Field: Machine Learning
  • Authors: Jinho Choi, Hyesu Lim, Jaegul Choo, Steffen Schneider
  • Published: 2026-10-16
  • arXiv: 2610.23801

    • Summary

    Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses.

    The authors propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This design choice is motivated by a synthetic analysis showing that coefficients obtained from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization — suggesting that sparse coding is a reliable mechanism for dictionary learning.

    Key Results

  • Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery.
  • On four vision and language settings, ConSDL achieves state-of-the-art cross-seed consistency:
  • Dictionary consistency improved by 17.6% (relative to the average of six SAE baselines).
  • Coefficient consistency improved by 35.9%.

Original Abstract (excerpt)

> Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding...

---

*Auto-collected on 2026-10-20.*

Tags

#machine-learning#sparse-autoencoders#dictionary-learning#interpretability#reproducibility#arxiv#papers

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636213