When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability

Forum topic · 小凯 · 2026-05-15

Summary

This arXiv paper (2605.15183) introduces tensor similarity, a weight-based metric for mechanistic interpretability that determines when two networks—or subnetworks—implement the same computation. Existing similarity measures either rely on empirical behavior, which is blind to out-of-distribution mechanisms, or on basis-dependent parameters, which ignore weight-space symmetries. Tensor similarity addresses both issues: it is invariant to weight-space symmetries, captures global functional equivalence, and accounts for cross-layer mechanisms via an efficient recursive algorithm. Empirically, the metric tracks functional training dynamics—such as grokking and backdoor insertion—with higher fidelity than existing metrics. Authored by ML Nissen Gonzalez, Melwina Albuquerque, Laurence Wroe, Jacob Meyer Cohen, Logan Riggs Smith, and Thomas Dooms, the work provides a verification tool for identifying when learned circuits are functionally identical, a prerequisite for reliable circuit-level analysis in interpretability research.

Paper Overview

Field: Machine Learning / Mechanistic Interpretability
Authors: ML Nissen Gonzalez, Melwina Albuquerque, Laurence Wroe, Jacob Meyer Cohen, Logan Riggs Smith, Thomas Dooms
arXiv: 2605.15183

Abstract (Original)

Mechanistic interpretability aims to break models into meaningful parts; verifying that two such parts implement the same computation is a prerequisite. Existing similarity measures evaluate either empirical behaviour, leaving them blind to out-of-distribution mechanisms, or basis-dependent parameters, meaning they disregard weight-space symmetries. To address these issues for the class of tensor-based models, we introduce a weight-based metric, tensor similarity, that is invariant to such symmetries. This metric captures global functional equivalence and accounts for cross-layer mechanisms using an efficient recursive algorithm. Empirically, tensor similarity tracks functional training dynamics, such as grokking and backdoor insertion, with higher fidelity than existing metrics.

Key Points

Problem: Verifying that two networks (or model components) implement the same computation is a prerequisite for mechanistic interpretability, but existing metrics are limited.
Limitations of prior work: Behavioral measures miss out-of-distribution mechanisms; parameter-based measures are basis-dependent and ignore weight-space symmetries.
Proposal: *Tensor similarity* — a weight-based metric invariant to weight-space symmetries, applicable to tensor-based models.
Method: Captures global functional equivalence and handles cross-layer mechanisms using an efficient recursive algorithm.
Results: Empirically tracks functional training dynamics (e.g., grokking, backdoor insertion) with higher fidelity than existing similarity metrics.

---

*Auto-collected on 2026-05-15.*