Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Differential Attention Fix

Forum topic · 小凯 · 2026-05-18

Summary

A forum post on zhichai.net discusses a systematic failure mode in Transformer-based continuous-time dynamic graph (CTDG) learning: attention dispersion. When temporal distribution shifts occur—for example, sudden changes in a social network's connection patterns around holidays—Transformer attention weights become overly uniform, assigning nearly equal scores to all historical neighbors regardless of relevance. Controlled ablation experiments identified a class of "critical nodes" carrying significantly stronger predictive signals than random neighbors, yet Transformers fail to focus attention on these nodes under temporal shift. The proposed fix applies differential attention, originally introduced for language modeling in the Differential Transformer: each attention head learns an offset vector subtracted from standard attention scores, preserving differentiated token-level signals while suppressing common-mode noise. Adding differential attention to three representative CTDG Transformer baselines consistently improved performance, with gains concentrated on high-shift datasets. Attention-level measurements confirmed the mechanism works—attention entropy decreased and attention mass on critical nodes increased. The authors built DiffDyG, combining differential attention with standard input encoding, achieving state-of-the-art results across 9 benchmarks under three negative sampling protocols. Open questions include the extra parameter cost, small gains on low-shift datasets, and lack of differentiation among node importance types.

Continuous-time dynamic graph (CTDG) learning is one of the harder problems in graph neural networks. Graph structure evolves over time—new friends join social networks, transaction relationships shift in financial networks—and models must capture these time-varying patterns. Transformer architectures now dominate this area, but Zhang, Zhao, Zeng, and Huang identified a systematic failure mode: attention dispersion.

The Problem: Attention Dispersion

When the temporal distribution of a graph shifts—for example, when a social network's connection patterns change drastically around holidays—Transformer attention weights become very uniform. Almost all historical neighbors receive similar attention scores, regardless of whether they are actually important.

Through controlled ablation experiments, the authors confirmed the existence of a class of critical nodes—nodes that carry significantly higher predictive signal than random neighbors. However, under temporal shift, Transformers fail to concentrate attention on these nodes, even when they are present in the input.

The Fix: Differential Attention

Instead of computing attention weights with a standard softmax, a "differential" operation is introduced in each attention head: each head learns an offset vector that is subtracted from the standard attention scores. This preserves differentiated token-level signals while suppressing common-mode noise.

This mechanism was previously used in language modeling (e.g., the Diff Transformer), but had never been applied to dynamic graphs.

Results

Adding differential attention to three representative CTDG Transformer baselines consistently improved performance across the board.
Gains were concentrated on high-shift datasets.
Attention-level measurements confirmed the mechanism is working: attention entropy decreased, and attention mass on critical nodes increased.

DiffDyG

Based on these findings, the authors built DiffDyG, a reference implementation combining differential attention with standard input encoding. It achieves state-of-the-art results on 9 benchmarks under three negative sampling protocols.

Open Questions

How many extra parameters does differential attention add?
Gains on low-shift datasets are small—are all "critical nodes" equally important? The current method does not distinguish between different types of node importance.

---

References

1. Zhang, J., Zhao, K., Zeng, Q., & Huang, L. (2026). *Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix*. arXiv:2605.16112 [cs.LG]. 2. Ye, H., et al. (2024). *Differential Transformer*. ICLR. 3. Kumar, S., et al. (2019). *Temporal Graph Networks for Deep Learning on Dynamic Graphs*. ICML.