You train two models with different architectures—a CNN and a Transformer—both on ImageNet classification. Both say "cat" for the same cat photo. But that only means they agree at the output layer: do they see the same "cat" in their intermediate layers?
This is not an abstract question. If you want to know whether two models truly behave similarly—whether their error patterns align, whether they respond identically to out-of-distribution data—you need to compare their internal representations. This comparison is critical in model merging, distillation, and transfer learning.
The most common tool is Centered Kernel Alignment (CKA): extract each layer's activations, compute a representation similarity matrix (RSM), and compare the two matrices. But Khandait and Gerken, in a recent paper (2605.15901), point out that CKA uses only a single resolution. Two models may look similar at one resolution and completely different at another.
Diffusion Geometry
Diffusion geometry is a framework popular in manifold learning. The core idea: define a random walk on the data—starting from each point, take a random step toward similar points—then observe the walk's long-time behavior.
If data points cluster tightly, the walk lingers within the cluster before escaping. If points are scattered in noise, the walk jumps erratically. Statistics of steps and jumps characterize the data's geometry across scales: short times (small scales) reveal intra-cluster structure; long times (large scales) reveal inter-cluster structure.
The key observation: an RSM—of the kind CKA uses—has an equivalence with a random-walk transition matrix, since an RSM can be row-normalized into a Markov matrix. Once converted, the entire diffusion-geometry toolbox becomes available for representation comparison.
Multi-Scale CKA
Concretely, they take powers of the transition matrix. At t=1, each step only reaches direct neighbors. Larger t lets you travel farther along the data manifold—the resolution shifts from "local" to "global." The parameter t acts like a magnification dial.
CKA scores at different t values can differ dramatically. Two models may be very similar locally (high CKA at t=1) yet differ greatly in global manifold structure (low CKA at t=10), or vice versa. Looking only at t=1 may miss global differences entirely.
From Layers to Networks
The authors go further. Existing methods mostly compare layer by layer (layer 3 to 5, layer 5 to 8), but nonlinear interactions across layers may be lost this way.
Their solution is alternating diffusion: fuse the Markov matrices of multiple layers into a single operator via alternating-diffusion techniques. This fused operator encodes sample geometry across multiple levels simultaneously, enabling direct network-to-network comparison instead of per-layer matching.
On the hardest ReSi benchmark—14 architectures, 7 datasets, 3 domains—their method achieves state-of-the-art on both language and vision tasks.
Open Questions
1. Fusion guarantees: How does alternating diffusion ensure the fused operator preserves each layer's independent geometry rather than destroying it? "Fusion" may imply information compression—under what conditions is the loss large? Unclear. 2. Choosing t: Multi-scale sounds good, but how should users pick which t values? Uniform sampling or adaptive selection? Too many t values increase computation; too few may miss key scales. The paper does a full sweep on ReSi but offers no simple guide for new users. 3. "Accuracy" of SoTA: The paper mentions SoTA "accuracy" and "output correlation," but it's unclear whether accuracy refers to CKA-score estimation error vs. ground truth or downstream task performance. The abstract doesn't distinguish these.
Still, the core idea is clean: representation comparison should not happen at a single scale. Diffusion geometry tells us how to systematize turning the magnification dial.
---
References
1. Khandait, A., & Gerken, J. E. (2026). *From Layers to Networks: Comparing Neural Representations via Diffusion Geometry*. arXiv:2605.15901 [cs.LG]. https://arxiv.org/abs/2605.15901 2. Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019). *Similarity of Neural Network Representations Revisited*. ICML 2019. 3. Coifman, R. R., & Lafon, S. (2006). *Diffusion Maps*. Applied and Computational Harmonic Analysis, 21(1), 5-30. 4. Lederman, R. R., & Talmon, R. (2018). *Learning the Geometry of Common Latent Variables Using Alternating-Diffusion*. Applied and Computational Harmonic Analysis, 44(3), 509-536. 5. Raghu, M., et al. (2017). *SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics*. NeurIPS 2017.