Train two BERT models with exactly the same data, architecture, and hyperparameters—the only difference being the random seed (42 vs. 123). Downstream performance is nearly identical. But open the black box with sparse autoencoders (SAEs), and an awkward fact emerges: the feature spaces don't line up.
Model A's "sentiment feature" lives at dimension 137; the same feature in Model B sits at 209. Model A expresses a "grammar feature" as a combination of three features; Model B uses five. They learn similar things but organize them completely differently—like two translation teams independently translating the same book with different chapter structures.
Bendegúz Váradi (Centre for Social Sciences, Budapest) and Zoltán Kmetty (Eötvös Loránd University) propose a solution: align the two models' activation spaces with a Procrustes rotation before training the SAE.
Why don't feature spaces align?
The root cause is the non-convexity of dictionary learning. SAEs learn a sparse dictionary where activations are represented by a few dictionary atoms, but the optimization has many local optima. Different seeds lead to different local optima, so the learned atoms occupy different positions and combinations.
This creates a fundamental challenge for mechanistic interpretability: without cross-model alignment, you can't claim a feature is universal—only that "Model A has this feature, Model B has that one."
Procrustes rotation: a mathematical "translator"
The Procrustes problem: given matrices A and B, find an orthogonal matrix R minimizing ||AR − B||. Intuitively, it rotates one point cloud onto another without stretching, preserving lengths and angles.
The pipeline: 1. Train two BERT models (different seeds) 2. Compute the Procrustes rotation matrix R between the two activation spaces 3. Rotate seed B's activation space into seed A's coordinate system 4. Train a joint SAE on the aligned activations
The full pipeline: Procrustes-conditioned Joint End-to-End Top-K SAE
Four components:
1. Procrustes alignment: orthogonal rotation between the two seeds' activation spaces 2. Top-K sparsity: keep only the K largest dictionary activations per vector (more stable than L1) 3. End-to-end optimization: the SAE is optimized jointly with a downstream task, so features are useful, not just reconstructive 4. Dead feature revival (AuxK loss): an auxiliary loss re-engages dictionary atoms that never activate
Experimental results
Tested on 5 independent seed pairs (10 BERT models) across 3 benchmarks:
- SST-2 (sentiment)
- Stanford Politeness
- TweetEval Emotion
- BERT-only: unknown whether Llama/GPT-scale models reach r ≥ 0.70
- Linear alignment only: Procrustes may be insufficient if feature spaces relate non-linearly
- Universality ceiling: even aligned, r = 0.70 not 1.0—the remaining ~0.30 is seed-specific, possibly reflecting each model's unique exploration path and local optima
- Framework: Apollo Research's e2e_sae — https://github.com/ApolloResearch/e2e_sae
- Paper: Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders
- Authors: Bendegúz Váradi, Zoltán Kmetty (Centre for Social Sciences, Budapest / Eötvös Loránd University)
- Date: July 9, 2026
The full pipeline achieves cross-seed Pearson correlation r ≥ 0.70, significantly outperforming post-hoc alignment baselines. If features were independent, r ≈ 0; if identical, r = 1.0. The 0.70 indicates substantial shared, universal features masked only by differing coordinate systems. Qualitatively, high-correlation features encode interpretable sociolinguistic patterns (sentiment words, politeness markers).
Why it matters
1. Reproducibility for mechanistic interpretability: feature discoveries become reproducible across seeds rather than artifacts of one training run. 2. Evidence for feature universality: independently trained models with identical architecture and data learn similar internal representations—differing only in coordinate systems. 3. Analogy to cross-lingual alignment: mathematically the same problem as cross-lingual word embedding alignment (MUSE, Vecmap)—two "dialects" of the same language rather than two different languages.