You train two models with different architectures—a CNN and a Transformer—both for ImageNet classification. Both say "cat" for the same cat photo. But agreement at the output layer doesn't mean they see the same thing in their intermediate layers.
This is not an abstract question. If you want to know whether two models behave similarly—their error patterns, their responses to out-of-distribution data—you need to compare their internal representations. This comparison is critical in model merging, distillation, and transfer learning.
The most common tool is Centered Kernel Alignment (CKA): take each layer's activations, compute a representation similarity matrix (RSM), and compare the matrices. But Khandait and Gerken, in a recent paper (arXiv:2605.15901), point out that CKA uses only a single resolution. Two models may look similar at one resolution and completely different at another.
Diffusion Geometry
Diffusion geometry is a framework popular in manifold learning. The core idea is simple: define a random walk over your data—from each point, take a random step toward similar points—then observe the long-time behavior of the walk.
If your points cluster tightly, the walk lingers within the cluster before escaping. If the points are scattered in noise, the walk jumps around. The statistics of these steps characterize the data's geometry at different scales: short times (small scale) reveal within-cluster structure; long times (large scale) reveal between-cluster structure.
The key observation: an RSM—the kind CKA uses—can be row-normalized into a Markov transition matrix. Once you do this conversion, the entire diffusion geometry toolbox becomes available for representation comparison.
Multi-Scale CKA
Concretely, they raise the transition matrix to the power t. At t=1, each step only reaches direct neighbors. At larger t, the walk travels farther along the data manifold—the resolution shifts from "local" to "global". The parameter t acts like a magnifier dial.
CKA scores at different t values can differ completely. Two models may be very similar in local representations (high CKA at t=1) while diverging in global manifold structure (low CKA at t=10), or vice versa. Looking only at t=1 may completely miss global differences.
From Layers to Networks
Most existing methods do layer-by-layer comparison: layer 3 vs layer 5, layer 5 vs layer 8. But a network operates as a whole, with nonlinear interactions across layers that pairwise comparison can miss.
Their solution is alternating diffusion: fuse the Markov matrices from multiple layers into a single operator using alternating-diffusion techniques. This fused operator encodes the sample geometry across multiple levels simultaneously, enabling direct network-to-network comparison in the fused space rather than layer-by-layer matching.
On the hardest ReSi benchmark—covering 14 architectures, 7 datasets, and 3 domains—their method achieves state-of-the-art results on both language and vision tasks.
Open Questions
1. Fusion and information loss. How does alternating diffusion guarantee the fused operator preserves each layer's independent geometric information rather than canceling it out? "Fusion" may imply compression. Under what conditions is the loss large? Unclear.
2. Choosing t. The parameter t becomes an extra hyperparameter. The paper says "multi-scale", but how should the set of t values be chosen—equally spaced or adaptively? Too many t values increase computation; too few may miss critical scales. The paper does a full sweep on ReSi but offers no simple guideline for new users.
3. What does "accuracy" mean? The paper mentions SoTA "accuracy" and "output correlation", but it's unclear whether "accuracy" refers to the accuracy of CKA scores themselves (estimation error against ground-truth similarity) or downstream task performance. The distinction matters, and the abstract doesn't clarify.
Still, the core idea is remarkably clean: representation comparison should not happen at a single scale. Diffusion geometry tells us how to systematize "turning the dial".
References
1. Khandait, A., & Gerken, J. E. (2026). *From Layers to Networks: Comparing Neural Representations via Diffusion Geometry*. arXiv:2605.15901 [cs.LG]. https://arxiv.org/abs/2605.15901 2. Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019). *Similarity of Neural Network Representations Revisited*. ICML 2019. 3. Coifman, R. R., & Lafon, S. (2006). *Diffusion Maps*. Applied and Computational Harmonic Analysis, 21(1), 5-30. 4. Lederman, R. R., & Talmon, R. (2018). *Learning the Geometry of Common Latent Variables Using Alternating-Diffusion*. Applied and Computational Harmonic Analysis, 44(3), 509-536. 5. Raghu, M., et al. (2017). *SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics*. NeurIPS 2017.