ICA Lens: Interpreting LLM Activations Without Training a Dictionary

Forum topic · 小凯 · 2026-06-12

Summary

A forum post on zhichai.net reviews the ICALens paper, which revives Independent Component Analysis (ICA), a classical signal processing method, as a training-free alternative to Sparse Autoencoders (SAEs) for LLM interpretability. SAEs require expensive overcomplete dictionaries (4x-64x hidden dimension) retrained per model, layer, and sparsity setting. ICALens instead maximizes non-Gaussianity of activation projections, since interpretable, selective directions are statistically less Gaussian. The authors introduce three stabilization techniques for FastICA on LLM activations: row normalization, p95 convergence fallback, and adaptive refitting. A novel diagnostic, Effective Receptive Field (ERF), shows shallow layers capture token-local features while deeper layers capture broad contextual ones, with kurtosis-ERF negatively correlated (Spearman -0.41 to -0.50). Human annotation labeled 142/150 sampled components confidently, and second-expert review supported 121/127. On SAEBench benchmarks, ICA matches SAEs on sparse probing and beats public SAEs on targeted probe perturbation at small-to-medium component budgets. The paper positions ICA as an efficient, complementary first lens rather than an SAE replacement, evaluated on GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B with open-source code, demos, and checkpoints.

Overview

Sparse Autoencoders (SAEs) dominate LLM interpretability, but they demand training overcomplete dictionaries (4x-64x hidden dimension) for every model, layer, and sparsity setting — a computational bottleneck that slows exploration. The ICALens paper asks a counterintuitive question: how much interpretable structure is already visible in activation geometry before training any neural dictionary?

Authors Sida Liu (independent researcher) and Feijiang Han (University of Maryland) revive Independent Component Analysis (ICA), a classical method from 1999, based on a key insight: interpretable directions are *selective* (activating only on specific tokens), and selective directions look statistically less Gaussian. ICA finds projections that maximize non-Gaussianity.

ICA was previously ignored in the LLM community due to numerically unstable off-the-shelf implementations and a lack of systematic auditing tools. ICALens is the first stable, efficient, auditable ICA workflow designed for LLMs.

Three Stabilization Recipes

1. Row normalization — LLM activations vary hugely in L2 norm, breaking ICA's whitening assumptions: x̃_i = x_i / max(||x_i||₂, ε). This improved strictly-converged layers on GPT-2 Small from 2 to 8 (6 to 10 after p95 fallback). 2. p95-LIM convergence fallback — accept a layer if 95% of components converge, flagging only unstable tail components instead of discarding the layer. 3. Adaptive refit — if full dimension d fails, retry at d/2, d/4, ... down to 16. Most layers accept the full dimension.

Effective Receptive Field (ERF): A Novel Diagnostic

For each component, ERF finds the shortest suffix length k where the component maintains same-signed activation in the top-15. Findings:

Shallow layers → token-local features (small ERF); deep layers → broad contextual features (large ERF)
Non-monotonic: middle layers have the most large-ERF components
Kurtosis-ERF negative correlation (Spearman -0.41 to -0.50): high kurtosis = small ERF = locally interpretable

This gives researchers a "microscope": start with high-kurtosis components, which are easier to understand.

Human Annotation

150 randomly sampled components (50 per model), independently annotated by two annotators:

| Model | High conf. | Medium | Low | Unclear | Labeled | |:---|:---|:---|:---|:---|:---| | GPT-2 Small | 44 | 4 | 1 | 1 | 49/50 | | Gemma 2 2B | 43 | 3 | 2 | 2 | 48/50 | | Qwen 3.5 2B | 40 | 1 | 4 | 5 | 45/50 | | Total | 127 | 8 | 7 | 8 | 142/150 |

A second expert reviewed the 127 high-confidence labels: 121 supported, 6 partially supported, 0 rejected; 112/127 scored ≥8/10. Label types: words (54), phrases (33), forms (11), sentences (20), global/long-range/positional/complex (25).

Quantitative Evaluation: ICA vs SAEs

Sparse Probing (SAEBench): ICA is competitive with public SAEs, and outperforms Matryoshka SAEs (small dictionaries), ITDA, and PCA.

Targeted Probe Perturbation (TPP): ICA beats public SAEs at small-to-medium component budgets; SAEs are competitive or better at large budgets. ICA's compactness (m ≤ d) is an advantage in tight-budget settings, while SAE overcompleteness pays off at scale.

ICA vs SAE: Complementary, Not a Replacement

| Property | ICA | SAE | |:---|:---|:---| | Training | None (analytic) | Gradient-based dictionary learning | | Dictionary size | Compact (m ≤ d) | Overcomplete (4x-64x d) | | Sparsity | Implicitly emergent | Explicitly penalized | | Objective | Non-Gaussianity maximization | Sparse reconstruction | | Direction type | Non-Gaussian, context-dependent | Sparse-activating, reconstruction-driven |

The two methods recover *correlated but non-redundant directions*: each captures features the other misses. The authors position ICA not as a "weak baseline" but as an efficient, complementary first lens — a fast way to decide which layers merit heavier dictionary learning.

Engineering

GPU-parallel FastICA built on FastICA_torch (PyTorch)
Symmetric (parallel) FastICA variant, log-cosh contrast function, PCA-based whitening, LIM convergence metric
Open-source: project page, online demo, HuggingFace checkpoints, GitHub code

Limitations

1. Capacity capped at hidden dimension (no overcompleteness like SAEs) 2. No explicit sparsity control 3. Some unstable tail components must be flagged 4. Not a substitute for high-resolution feature discovery

The broader takeaway: the interpretability community may be stuck in an arms race of ever-larger dictionaries. ICALens reminds us to inspect the data's own geometry before stacking more neural networks — sometimes a 1999 signal processing method is more direct than a 2024 deep learning one.

References

Liu, S. & Han, F., "ICA Lens: Interpreting Language Models Without Training Another Dictionary", arXiv:2606.11722, 2026
Models evaluated: GPT-2 Small, Gemma 2 2B, Qwen 3.5 2B Base
Benchmarks: SAEBench (Sparse Probing, Targeted Probe Perturbation); dataset: Pile-10k (1M activations)