Can AI Weather Models Predict Beyond Two Weeks? ETH Zurich Benchmarks 9 Models and Finds Three Failure Modes

Forum topic · 小凯 · 2026-05-31

Summary

Researchers at ETH Zurich and the University of Cambridge benchmarked nine leading AI weather models—including Pangu, GraphCast, FourCastNet, Aurora, SFNO, AIFS, and DLESyM—on autoregressive rollouts of up to two years (14,600 steps). Only three models completed the run. The study is the first to systematically classify long-horizon failures into three modes: blow-up (exponential divergence, as with FourCastNet at ~8 days and GraphCast at ~300 days), drift (gradual seasonal temperature shifts), and loss of seasonality (Pangu freezing after ~180 days, traced to its lack of time embedding). Noise-injection experiments show stable models act as denoisers that converge to a climate attractor—Aurora even recovered a physical seasonal cycle from pure white noise or a cat photo—without memorizing training data. Ablations on small Aurora variants reveal that coarser spatiotemporal resolution (1.5°, 24-hour steps) improves stability, while only time embedding controls seasonality. Decadal runs show stable models capture the shape of extreme temperature events but systematically underestimate extreme heat. The work exposes an inherent tension between short-term forecast precision and long-term climate-simulation stability.

Paper: Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts Authors: Fanny Lehmann, Firat Ozdemir, Yun Cheng, Torsten Hoefler, Sebastian Schemm, Benedikt Soja, Siddhartha Mishra Institutions: ETH Zurich (AI Center, SDSC, SPCL, IGP), University of Cambridge arXiv ID: 2605.30184 Submitted: May 28, 2026 | Categories: cs.LG, physics.ao-ph

Key Points

Setup: Nine top AI weather models (Pangu, GraphCast, FourCastNet, Aurora, SFNO, AIFS, DLESyM, etc.) were run autoregressively for up to two years—14,600 six-hour steps. Only three models survived the full run; six failed along the way.
Three failure modes (the first systematic classification of long-rollout failures):
Blow-up: exponential divergence from physical reality. FourCastNet collapsed after ~8 days; GraphCast held about 300 days before some variables diverged exponentially.
Drift: no explosion, but gradual bias—seasonal temperatures slowly shift, compounding year over year.
Loss of seasonality: the subtlest mode. Pangu's global temperature forecasts "froze" after ~180 days into a time-invariant state. The direct cause: Pangu has no time embedding, so it cannot know what month it is. Design lesson: to run beyond a year, tell the model the date.
Stable models are denoisers. Injecting Gaussian noise shows Aurora and SFNO progressively filter noise toward physically plausible states, while GraphCast and AIFS amplify it. Given pure white-noise input, Aurora generated physically plausible weather trajectories with a seasonal cycle—each initialization producing a unique trajectory.
Not memorization. Distance-ratio tests show that in year-long rollouts, Aurora and SFNO outputs are nearly equidistant to the first- and second-nearest training samples—the opposite of what retrieval would produce.
The climate attractor. Initializing Aurora with a cat photo (pixels mapped to atmospheric variables), the model denoised it into a physically plausible atmosphere within days, converging toward a realistic seasonal trajectory—an implicit attractor toward Earth's climate manifold.
Ablations (AuroraS, 113M parameters): Attention window size, window shifting, patch size, LayerNorm vs RMSNorm, number of vertical levels, and static variables do not affect stability. Time embedding controls seasonality only. What determines stability is spatiotemporal resolution: 1.5° / 24-hour-step models are more stable than 0.25° / 6-hour ones, since fine resolution introduces high-frequency noise that autoregression amplifies. This explains why climate emulators (coarser scales) are generally more stable than forecast models.
Extremes: right shape, wrong amplitude. Ten-year runs of Aurora, SFNO, and DLESyM reproduce extreme temperature events across five regions, but all systematically underestimate extreme heat (hottest days several degrees cooler than ERA5) and SFNO overestimates extreme cold in some regions. Stable models capture the statistics' shape but smooth the magnitudes—problematic for climate research, which cares most about extremes.

What the Paper Does Not Answer

Why graph-based architectures (GraphCast, AIFS) amplify high-frequency noise while SFNO and 3D Swin Transformers stay stable—no theoretical mechanism given.
Robustness under climate-change distribution shift; all models were trained on 1979–2020 data.
The mechanism behind longer time steps (24h) being more stable than shorter ones (6h).
The origin of systematic hot/cold extreme biases (insufficient training extremes vs. mean-seeking denoising).

Takeaway

AI weather forecasting is splitting into two paths: replacing numerical solvers for fast short-term forecasts (demanding precision, fine scales, high-frequency detail) and replacing climate models for decadal simulation (demanding stability, coarse scales, low-frequency fidelity). The paper's central message is that these two goals are in tension—improving short-term resolution undermines long-horizon stability, and vice versa. The line between the two is drawn by how a model handles high-frequency energy at small scales: suppress it and it survives; amplify it and it collapses. Future AI climate models will have to cross that line.

References: 1. Lehmann et al., "Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts", arXiv:2605.30184, 2026. 2. Bodnar et al., "Aurora: A Foundation Model for the Earth System", Nature, 2025. 3. Bonev et al., "Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere", ICML 2023. 4. Kochkov et al., "Neural General Circulation Models for Weather and Climate", Nature, 2024. 5. Bi et al., "Pangu-Weather: Accurate Medium-Range Global Weather Forecasting with 3D Neural Networks", Nature, 2023.