DiffDyG: Diagnosing and Fixing Attention Dispersion in Dynamic Graph Transformers with Differential Attention

Forum topic · 小凯 · 2026-05-18

Summary

Continuous-time dynamic graph (CTDG) learning requires models to capture time-varying structural patterns in networks such as social and financial graphs. Zhang, Zhao, Zeng, and Huang identify a systematic failure mode in Transformer-based CTDG models: attention dispersion. When temporal distribution shift occurs—for example, sharp changes in connection patterns around holidays—attention weights become nearly uniform, assigning similar scores to all historical neighbors regardless of importance. Controlled ablation experiments reveal a class of 'key nodes' carrying significantly higher predictive signal than random neighbors, which Transformers fail to attend to under shift. The proposed fix applies differential attention: each attention head learns an offset vector subtracted from standard attention scores, suppressing common-mode noise while retaining differentiated token-level signals. Adding this mechanism to three representative CTDG Transformer baselines consistently improves performance, with gains concentrated on high-shift datasets. Attention-level measurements confirm the mechanism: attention entropy decreases and attention mass on key nodes increases. The authors build DiffDyG, a reference implementation combining differential attention with standard input encoding, achieving state-of-the-art results across 9 benchmarks and three negative sampling protocols. Open questions include the parameter overhead, the small gains on low-shift datasets, and the lack of differentiation among node importance types.

Continuous-time dynamic graph (CTDG) learning is one of the harder problems in graph neural networks. Graph structure changes over time—new friends join social networks, trading relationships shift in financial networks—and models must capture these time-varying patterns. Transformer architectures have come to dominate this domain, but Zhang, Zhao, Zeng, and Huang identify a systematic failure mode: attention dispersion.

The Problem: Attention Dispersion

When the temporal distribution of a graph shifts—for example, drastic changes in a social network's connection patterns around holidays—Transformer attention weights become very uniform. Nearly all historical neighbors receive roughly the same attention scores, regardless of whether they are actually important.

Through controlled ablation experiments, the authors confirm the existence of a class of key nodes—nodes carrying significantly higher predictive signal than random neighbors. Under temporal shift, however, the Transformer fails to concentrate attention on these nodes, even though they are present in the input.

The Fix: Differential Attention

Instead of computing attention weights with a standard softmax, the method introduces a differential operation within each attention head: each head learns an offset vector that is subtracted from the standard attention scores. This preserves differentiated token-level signals while suppressing common-mode noise.

This mechanism was previously used in language modeling (e.g., the Diff Transformer), but had never been applied to dynamic graphs.

Results

Adding differential attention to three representative CTDG Transformer baselines consistently improves performance across the board.
Gains are concentrated on high-shift datasets.
Attention-level measurements confirm the mechanism is working: attention entropy decreases, and attention mass on key nodes increases.

Based on this finding, the authors build DiffDyG, a reference implementation combining differential attention with standard input encoding. It achieves state-of-the-art results on 9 benchmarks under three negative sampling protocols.

Open Questions

What is the additional parameter count introduced by differential attention?
Gains are small on low-shift datasets—does that mean not all "key nodes" are equally important?
The current method does not distinguish between different types of node importance.

References

1. Zhang, J., Zhao, K., Zeng, Q., & Huang, L. (2026). *Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix*. arXiv:2605.16112 [cs.LG]. 2. Ye, H., et al. (2024). *Differential Transformer*. ICLR. 3. Kumar, S., et al. (2019). *Temporal Graph Networks for Deep Learning on Dynamic Graphs*. ICML.