Train two BERT models with exactly the same data, architecture, and hyperparameters—only the random seed differs (42 vs. 123). Their downstream performance is nearly identical. But when you extract features with sparse autoencoders (SAEs), an awkward fact emerges: their feature spaces don't line up.
Model A's "sentiment feature" sits at dimension 137; Model B's equivalent sits at 209. Model A composes "syntax" from three features, Model B from five. They learned similar things, but organized them completely differently—like two translation teams independently translating the same book, one dividing by chapters, the other by topic, with page numbers that never match.
A July 2026 paper by Bendegúz Váradi (Centre for Social Sciences, Budapest) and Zoltán Kmetty (Eötvös Loránd University) proposes a fix: align the two models' activation spaces with a Procrustes rotation before training the SAE.
Why don't feature spaces align?
The root cause is the non-convexity of dictionary learning. SAEs train a sparse dictionary on layer activations so each activation is a linear combination of a few dictionary atoms. Because the optimization problem has many local optima, different seeds lead to different arrangements of dictionary atoms. Both models learn "sentiment," "syntax," and "topic" features, but at different positions with different compositions.
This poses a fundamental challenge for mechanistic interpretability: if you can't align features across models, you can't claim a feature is universal—only that model A has one feature and model B has another.
Procrustes rotation: a mathematical translator
The Procrustes problem finds an orthogonal matrix R minimizing ||AR − B|| for given matrices A and B. Intuitively, it rotates one shape onto another while preserving vector lengths and angles (no stretching or compression).
For cross-seed SAEs: 1. Train two BERT models with different seeds 2. Compute a Procrustes rotation matrix R between the two activation spaces 3. Rotate seed B's activation space into seed A's coordinate system 4. Jointly train a SAE on the aligned activations
The resulting SAE's dictionary atoms have comparable activation patterns across both models.
The full pipeline
The paper's pipeline is a Procrustes-conditioned Joint End-to-End Top-K SAE with four components:
1. Procrustes alignment: orthogonal rotation between the two seeds' activation spaces 2. Top-K sparsity: keep only the K largest dictionary-atom responses per activation (more stable than L1) 3. End-to-end optimization: the SAE is optimized jointly with a downstream task, so features are useful, not just reconstructive 4. Dead feature revival (AuxK loss): an auxiliary loss re-engages dictionary atoms that never activate
Experimental results
Tested on 5 independent seed pairs (10 BERT models) across 3 benchmarks—SST-2 (sentiment), Stanford Politeness, and TweetEval Emotion—the full pipeline yields features with cross-seed Pearson correlation r ≥ 0.70, significantly outperforming post-hoc alignment baselines.
For context: r ≈ 0 would mean unrelated features; r = 1.0 would mean identical features. At 0.70, a substantial fraction of features are genuinely universal—both models learn similar things, previously masked by their coordinate systems. Qualitative analysis confirms that highly universal features encode interpretable sociolinguistic patterns (sentiment words, politeness markers, etc.).
Why it matters
1. Reproducibility for mechanistic interpretability: if feature spaces aren't comparable across runs, the claim "the model learned feature X" has no general meaning. Procrustes alignment makes feature discovery reproducible across models. 2. Evidence for feature universality: r ≥ 0.70 between independently trained models supports the hypothesis that models with the same architecture and data learn similar internal representations—just in different coordinate systems. 3. Analogy to cross-lingual alignment: this is mathematically the same problem as cross-lingual embedding alignment (MUSE, Vecmap)—two feature spaces expressing the same semantics with different "vocabularies." The difference: cross-seed alignment is between two dialects of the same language—smaller gaps, but still non-trivial.
Limitations
- Only BERT tested: whether larger models (Llama, GPT-scale) reach r ≥ 0.70 is unknown
- Procrustes is linear: if feature spaces relate non-linearly (rotation + scaling), linear Procrustes may be insufficient
- A universality ceiling: even aligned, r is 0.70, not 1.0. The remaining ~0.30 is seed-specific—each model learned things the other didn't, likely reflecting stochastic exploration paths to different local optima. This "personality" of each model is itself an interesting object of study.
Open-source code
The paper uses Apollo Research's e2e_sae framework: https://github.com/ApolloResearch/e2e_sae
---
Paper: Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders Authors: Bendegúz Váradi, Zoltán Kmetty (Centre for Social Sciences, Budapest / Eötvös Loránd University) Code: https://github.com/ApolloResearch/e2e_sae Date: July 9, 2026