Distilling Graph Geometry: Closing the Knowledge Gap from GNNs to MLPs

Forum topic · 小凯 · 2026-10-09

Summary

A new arXiv paper (2610.10520) by Zhewei Chen, Hao Zhu, Jiaojiao Jiang, and Ahad N. Zehmakan addresses GNN-to-MLP distillation, where a message-passing graph neural network teacher transfers its accuracy to a graph-free MLP student for inference. The authors identify two spectral failure modes in existing methods that only transfer node-wise predictions or use confidence-based reweighting: on sparse graphs, students suffer spectral underfit, missing high-energy teacher directions concentrated near boundary regions; on dense graphs, students suffer spectral overfit, retaining spurious directions already collapsed by the teacher's aggregation. They propose G2MLP (Graph Geometry-aware MLP), a training-time distillation framework guided by Ollivier-Ricci curvature, which localizes where these spectral errors occur and allocates supervision between prediction-level and representation-level alignment. The deployed model remains a standard MLP requiring no graph at inference. Experiments on node classification benchmarks show consistent improvements over graph-free distillation baselines, reduced teacher-student rank gap in both regimes, and transferability to Graph Transformer teachers and link prediction tasks without architectural changes.

Paper Overview

Field: Machine Learning Authors: Zhewei Chen, Hao Zhu, Jiaojiao Jiang, Ahad N. Zehmakan Published: 2026-10-07 arXiv: 2610.10520

Abstract

GNN-to-MLP distillation aims to retain the predictive accuracy of a message-passing teacher while deploying a graph-free MLP at inference. Existing methods mainly transfer node-wise predictions or use confidence-based reweighting, but they do not specify where the student should preserve the teacher's graph-induced geometry.

We show that this omission leads to two spectral failure modes in the student's representation space. On sparse graphs, the student suffers from spectral underfit, missing high-energy teacher directions concentrated near boundary regions. On dense graphs, it suffers from spectral overfit, retaining spurious directions that the teacher has collapsed through aggregation.

Motivated by an energy-weighted teacher-student alignment objective, we propose Graph Geometry-aware MLP (G²MLP), a training-time distillation framework guided by Ollivier-Ricci curvature. Curvature identifies where the two spectral errors concentrate, used to allocate supervision between prediction-level and representation-level alignment. The deployed model remains a standard MLP with no graph access at inference.

On node classification benchmarks, G²MLP consistently outperforms graph-free distillation baselines, reduces the teacher-student rank gap in both regimes, and transfers to Graph Transformer teachers and link prediction tasks without architectural modifications.

Key Contributions

Identification of two spectral failure modes (spectral underfit on sparse graphs, spectral overfit on dense graphs) in GNN-to-MLP distillation
G²MLP: an Ollivier-Ricci curvature-guided training-time distillation framework
Graph-free inference: deployed model is a standard MLP
Generalization to Graph Transformer teachers and link prediction without architecture changes