How Distribution Shift Shapes Pretraining Gains in Neural PDE Surrogates

Forum topic · 小凯 · 2026-09-19

Summary

This arXiv paper (2609.20814) investigates how different components of distribution shift affect the value of pretraining neural PDE surrogate models for CFD tasks. The authors pretrain a surrogate on 254,909 RANS solutions from one airfoil family and fine-tune it on a new family under two target settings with matched freestream ranges: the same Spalart-Allmaras (SA) turbulence model, and SA augmented with e^N transition modeling. At a data budget of N=1000, the pretrained model matches the accuracy of a from-scratch model trained on 3.25x more samples for the same-SA target, and 2.58x more samples for the transition-modeled target. However, by N=5000 the ordering reverses (1.56x versus 1.86x), showing that pretraining benefits diminish with larger target budgets. Additionally, sampling more distinct airfoils during pretraining reduces error on both targets at N=1000, but the gain exceeds observed inter-run variance only for the same-SA target. The findings indicate that pretraining value is jointly determined by the target data budget, target data coverage, and whether source and target differ in modeled physics.

Paper Overview

Field: Machine Learning / Scientific ML
Authors: Pochinapeddi Sai Bhargav, Nithin Somasekharan, Rohit Sunil Kanchi, Sicheng He, Shaowu Pan
Published: 2026-09-17
arXiv: 2609.20814

Abstract

Pretraining a neural PDE surrogate can reduce the amount of new CFD data needed when geometry or modeled physics changes. However, it remains unclear how different components of distribution shift affect this benefit. The authors pretrain a surrogate on 254,909 RANS solutions from one airfoil family and fine-tune it on a new family under two target settings with matched freestream ranges: the same Spalart-Allmaras (SA) turbulence modeling, and SA with added e^N transition modeling.

Key Findings

At N=1000, the pretrained model matches the accuracy of a model trained from scratch on 3.25x as many samples for the same-SA target, and 2.58x as many for the transition-modeled target.
By N=5000, this ordering reverses (1.56x versus 1.86x), showing that pretraining gains depend heavily on the target data budget.
At N=1000, sampling more distinct airfoils during pretraining lowers error on both targets, but the gain only exceeds observed inter-run variance (a 3.3x to 4.0x range) for the same-SA target.
Overall, the value of pretraining is jointly determined by three factors: the target data budget, the coverage of target data, and whether source and target differ in modeled physics.

Takeaway

For practitioners using neural PDE surrogates in aerodynamic design, pretraining is most valuable when the target dataset is small and the underlying physics matches the pretraining source; gains can shrink or even reverse as target data grows or when transition physics changes.