AI Weather Models Beyond Two Weeks: Benchmarking 9 Models and Three Failure Modes

Forum topic · 小凯 · 2026-05-31

Summary

A forum post discusses a benchmark study from ETH Zurich and the University of Cambridge testing nine leading AI weather models, including Pangu-Weather, GraphCast, FourCastNet, Aurora, SFNO, and AIFS, on autoregressive rollouts of up to two years (14,600 steps). The study identifies three failure modes: blow-up (exponential divergence, e.g., FourCastNet after ~8 days), drift (gradual bias accumulation), and loss of seasonality (Pangu freezing after ~180 days due to lacking time embedding). Aurora, SFNO, and DLESyM completed the full two-year rollout. The analysis shows stable models act as denoisers: they suppress high-frequency noise and converge toward a physically plausible climate attractor—Aurora even recovered a seasonal cycle when initialized with pure white noise or an image of a cat. Ablation experiments with small Aurora variants indicate stability is governed mainly by spatiotemporal resolution (coarser resolution and longer steps are more stable), not attention windows or normalization layers. Stable models captured the frequency distribution of extreme weather events but systematically underestimated heat extremes. The post highlights a fundamental trade-off between short-range accuracy and long-range stability, with implications for AI-based climate modeling.

This post reviews a quantitative benchmark study on whether AI weather models can predict beyond the well-known two-week limit of weather forecasting.

Key points

The experiment

ETH Zurich researchers ran 9 top AI weather models (Pangu-Weather, GraphCast, FourCastNet, Aurora, SFNO, AIFS, DLESyM, etc.) in autoregressive rollouts of up to two years — 14,600 steps at 6-hour intervals.
Only three models (Aurora, SFNO, DLESyM) completed the full rollout; six failed along the way.

Three failure modes

*Blow-up*: exponential divergence from physical reality. FourCastNet collapsed after ~8 days; GraphCast diverged after ~300 days.
*Drift*: slow bias accumulation — seasonal structure persists, but temperatures shift year over year.
*Loss of seasonality*: Pangu "froze" after ~180 days, producing a time-invariant climate. The cause: Pangu has no time embedding, so it cannot know the calendar date and loses the seasonal signal over long rollouts.

Stable models are denoisers

Adding Gaussian noise to inputs showed Aurora and SFNO progressively remove noise, converging toward physically plausible states — while GraphCast and AIFS amplify noise.
Aurora produced seasonal, physically realistic weather trajectories from pure white-noise initialization, and even "denoised" an image of a cat mapped onto atmospheric variables back into winter — evidence of an implicit climate attractor.
Distance-ratio tests show outputs are not memorized training samples: nearest and second-nearest training neighbors are nearly equidistant.

Ablation findings

Attention window size, patch size, normalization type, number of vertical levels, and static variables do not determine stability.
Time embedding controls seasonality; removing it removes seasons but not stability.
Spatiotemporal resolution is the key stability factor: coarse resolution (1.5°, 24-hour steps) is more stable than fine resolution (0.25°, 6-hour steps), because fine scales contain high-frequency energy that amplifies during autoregression.

Extreme weather

Ten-year rollouts captured the shape of extreme-event statistics but not magnitudes: Aurora and DLESyM underestimated extreme heat by several degrees; SFNO overestimated extreme cold in some regions.

Implications

The study frames a fundamental trade-off: short-range precision (fine resolution, short steps) reduces long-range stability, and vice versa — the same tension exists in classical numerical models, but AI models make it visible. AI models trained on historical data may also face untested distribution shifts under climate change. The post argues that AI weather forecasting and AI climate emulation are diverging into two distinct technical paths with different requirements.

References

1. Lehmann et al., "Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts", arXiv:2605.30184, 2026. 2. Bodnar et al., "Aurora: A Foundation Model for the Earth System", Nature, 2025. 3. Bonev et al., "Spherical Fourier Neural Operators", ICML 2023. 4. Kochkov et al., "Neural General Circulation Models for Weather and Climate", Nature, 2024. 5. Bi et al., "Pangu-Weather", Nature, 2023.