ICA Lens: Interpreting LLM Activations Without Training a Dictionary
*A 40-year-old signal processing method regains prominence for LLM interpretability.*
Background: The tyranny of SAEs
The standard tool for LLM interpretability is the Sparse Autoencoder (SAE), but it carries heavy costs:
- Every layer requires training an overcomplete dictionary (4x–64x hidden dimension);
- Each model, layer, and sparsity setting needs retraining;
- Training, storing, and evaluating these dictionaries becomes a computational bottleneck for rapid exploration.
- Layer-wise transfer: shallow layers → token-local (small ERF); deep layers → broad context (large ERF);
- Non-monotonic: middle layers have the most large-ERF components;
- Kurtosis–ERF negative correlation (Spearman -0.41 to -0.50): high kurtosis = small ERF = locally interpretable; low kurtosis = large ERF = context-dependent.
- Small-to-medium component budgets: ICA beats public SAEs;
- Large budgets: SAEs compete or win.
- Built on FastICA_torch (PyTorch GPU);
- Symmetric (parallel) FastICA variant;
- Log-cosh contrast function;
- PCA-based whitening;
- LIM convergence metric: ||W_new − W_old||;
- Fully GPU-parallel linear algebra.
- Liu, S. & Han, F., "ICA Lens: Interpreting Language Models Without Training Another Dictionary", arXiv:2606.11722, 2026
- Affiliations: Independent researcher / University of Maryland
- Models evaluated: GPT-2 Small, Gemma 2 2B, Qwen 3.5 2B Base
- Benchmarks: SAEBench (Sparse Probing, Targeted Probe Perturbation)
- Dataset: Pile-10k (1M activations)
The paper asks a counterintuitive question: how much interpretable structure is already visible in activation geometry before training another neural dictionary?
ICA's revival: finding signal in non-Gaussian directions
Authors Sida Liu (independent researcher) and Feijiang Han (UMD) return to Independent Component Analysis (ICA) from 1999.
Core insight: interpretable directions should be *selective*—activating only on specific tokens—and selective directions look less Gaussian statistically. ICA's classical objective is exactly to find directions whose projections are maximally non-Gaussian:
ICA was previously ignored by the LLM community because off-the-shelf implementations were numerically unstable and lacked systematic inspection, labeling, and evaluation tooling. ICALens is the first stable, efficient, auditable ICA workflow designed for LLMs.
Three stabilization recipes
1. Row normalization: LLM activations vary wildly in ℓ₂ norm, breaking standard ICA assumptions. Normalize as x̃_i = x_i / max(||x_i||₂, ε). Strictly converged layers on GPT-2 Small rose from 2 → 8 (6 → 10 after p95 fallback). 2. p95-LIM convergence fallback: If 95% of components are stable (p95-LIM ≤ τ), accept the layer; only flag unstable tail components. 3. Adaptive refit: If full dimension d fails to converge, try d/2, d/4, ... down to 16. Most layers accept the full dimension.
Effective Receptive Field (ERF): a novel diagnostic
For each ICA component, find the shortest suffix length k where the component stays top-15 with the same-sign activation.
Findings (Figures 4–5):
This provides a "microscope" for interpretability: look at high-kurtosis components first, as they are easier to understand.
Human annotation: 142/150 components labeled
150 randomly sampled components (50 per model), two independent annotators:
| Model | High conf. | Medium | Low | Unclear | Label rate | |:---|:---|:---|:---|:---|:---| | GPT-2 Small | 44 | 4 | 1 | 1 | 49/50 | | Gemma 2 2B | 43 | 3 | 2 | 2 | 48/50 | | Qwen 3.5 2B | 40 | 1 | 4 | 5 | 45/50 | | Total | 127 | 8 | 7 | 8 | 142/150 |
Second expert review of the 127 high-confidence labels: 121 supported, 6 partially supported, 0 rejected. 112/127 rated ≥8/10.
Label types: words (54), phrases (33), forms (11), sentences (20), global/long-range/positional/complex (25).
Quantitative evaluation: ICA vs SAE
Experiment 1 — Sparse Probing (SAEBench): ICA competes with public SAEs; Matryoshka SAE (small dictionary), ITDA, and PCA all fall below ICA.
Experiment 2 — Targeted Probe Perturbation (TPP):
ICA's compactness (m ≤ d) is an advantage at small budgets, while SAE's overcompleteness covers more features at large budgets.
Complementary, not a replacement
| Property | ICA | SAE | |:---|:---|:---| | Training | None (analytic) | Gradient-based dictionary learning | | Dictionary size | Compact (m ≤ d) | Overcomplete (4x–64x d) | | Sparsity | Implicitly emergent | Explicit penalty | | Objective | Non-Gaussianity maximization | Sparse reconstruction | | Direction type | Non-Gaussian, context-dependent | Sparse-activating, reconstructive |
Key insight: ICA and SAE recover related but non-redundant directions. ICA finds non-Gaussian, context-dependent directions; SAEs find sparsely activating, reconstruction-optimized directions. Each captures things the other misses.
The authors' position is clear: ICA should not be viewed as a "weak baseline" but as an "efficient and complementary first lens"—helping you quickly decide which layers deserve heavier dictionary learning.
Engineering: GPU-parallel FastICA
The project is open-sourced: project page, online demo, HuggingFace checkpoints, and GitHub code.
Limitations and reflections
1. Capacity ceiling: ICA component count is fixed by hidden dimension (vs. SAE overcompleteness); 2. No explicit sparsity: sparsity emerges from non-Gaussianity, less controllable than SAEs; 3. Unstable tails: some components must be flagged as unreliable; 4. Not a replacement for high-resolution exploration: SAEs remain necessary for fine-grained feature discovery.
The broader takeaway: the interpretability community risks an arms race of ever-larger dictionaries. ICALens reminds us to examine the data's own geometry before stacking more neural networks. Sometimes a 1999 signal processing method is more direct than 2024 deep learning.