Why Adversarial Training Improves PINNs: A Neural Tangent Kernel Perspective

Forum topic · 小凯 · 2026-05-18

Summary

Physics-informed neural networks (PINNs) often fail to learn high-frequency, multi-scale PDE solutions due to spectral bias, sometimes collapsing to trivial zero solutions. Recent work shows that adversarial training with a GAN-style discriminator can significantly improve PINN training, but the mechanism was unclear. Drawing on the Cao, So, Wang, and Wang analysis, this post explains the effect through the neural tangent kernel (NTK): the PINN's NTK has large eigenvalues for low-frequency components and near-zero eigenvalues for high-frequency ones, so high-frequency modes are barely learned. The discriminator dynamically reweights frequency components—amplifying gradient signals on high-frequency errors—effectively reshaping the NTK eigenvalue spectrum to make learning rates more balanced across frequencies. Based on this theory, the authors propose an efficient training algorithm that selectively perturbs PINN predictions to emulate the discriminator, avoiding full GAN training. Open questions remain: NTK analysis holds strictly in the infinite-width limit, stability gains may not transfer uniformly across PDE types, and claims of orders-of-magnitude accuracy improvements need qualification. Includes references to the arXiv paper, the original PINNs work by Raissi et al., and Jacot et al.'s NTK paper.

Physics-informed neural networks (PINNs) approximate PDE solutions by including both data-fitting and physics-residual terms in the loss. In practice, however, PINNs are hard to train: high-frequency, multi-scale solutions are barely learned, and models often converge to trivial zero solutions. This is spectral bias — neural networks inherently favor low-frequency functions, and learning of high-frequency components stalls.

Recent work found that GAN-style adversarial training significantly improves PINN training quality — the discriminator provides more informative gradient signals to the generator (the PINN). But nobody knew why.

The NTK Explanation

Cao, So, Wang, and Wang provide a theoretical explanation via the neural tangent kernel (NTK). The NTK governs how fast different frequency components of a network's output are learned during training:

The PINN's NTK has a huge eigenvalue gap between low- and high-frequency regions.
Low-frequency components have large eigenvalues → learned quickly.
High-frequency components have near-zero eigenvalues → essentially not learned.

What Adversarial Training Does

The discriminator dynamically reweights frequency components: when the PINN fits only the low-frequency part, the discriminator produces a gradient signal that amplifies high-frequency errors. Crucially, this does not modify the loss function — it changes the eigenvalue distribution of the NTK during training dynamics, making the PINN's NTK more balanced so high-frequency components get larger learning rates.

A Practical Algorithm

Based on this analysis, the authors propose a new efficient training algorithm that avoids full GAN training — it only selectively perturbs the PINN's predictions to emulate the discriminator's effect.

Open Questions

The NTK analysis holds strictly in the infinite-width limit; for small practical PINNs, the NTK approximation may deviate.
Does the added stability from adversarial training hold uniformly across all PDE types?
The paper claims accuracy improvements of "several orders of magnitude" — on which problems does this hold, and on which does it fail?

---

References

1. Cao, Y., So, C. C., Wang, J., & Wang, H. (2026). *When and Why Adversarial Training Improves PINNs: A Neural Tangent Kernel Perspective*. arXiv:2605.15959 [cs.LG]. 2. Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). *Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations*. Journal of Computational Physics. 3. Jacot, A., Gabriel, F., & Hongler, C. (2018). *Neural Tangent Kernel: Convergence and Generalization in Neural Networks*. NeurIPS.