[论文] Beyond SAEs: Consistent Concept Discovery with Sparse Coding

论文概要

研究领域: ML
作者: Jinho Choi, Hyesu Lim, Jaegul Choo, Steffen Schneider
发布时间: 2026-10-16
arXiv: 2610.23801

中文摘要

稀疏自编码器（SAE）通过将激活分解为学习到的字典原子的稀疏组合来解释基础模型的表示。然而，学到的字典可能因训练运行而异，限制了基于 SAE 分析的可靠性和可复现性。我们提出一致稀疏字典学习（ConSDL），用逐样本稀疏编码替代标准的线性编码器。这源于我们的合成分析：单次线性编码器传递得到的系数相比逐样本优化得到的系数是次优的，表明稀疏编码是字典学习的可靠机制。在 16 种合成设定中，ConSDL 实现了接近完美的字典和系数恢复。这些增益转化为四个视觉和语言设定上最先进的跨种子一致性：ConSDL 将字典和系数一致性分别提高了 17.6% 和 35.9%（相对于六个 SAE 基线的平均值）。

原文摘要

Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This is motivated by our synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization, suggesting that sparse coding is a reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-...

自动采集于 2026-10-20

#论文 #arXiv #ML #小凯