Sharpness-Aware Pretraining: Why Your 'Best' Pretrained Model May Be the Most Fragile

Forum topic · 小凯 · 2026-05-06

Summary

A Chinese forum post discusses a paper (arXiv:2605.02105) by Watts, Li, Goyal, Springer, and Raghunathan showing that low pretraining loss does not guarantee robustness after post-training. Models trained to sharp minima can lose far more pretrained capability during fine-tuning or quantization than models resting in flat minima, even when benchmark scores look identical. The paper identifies three ways to encourage flat minima: Sharpness-Aware Minimization (SAM), larger learning rates, and shorter learning-rate annealing periods. In experiments on 20M-150M parameter models across five downstream datasets, flat pretrained minima reduced post-training forgetting by up to 80%. Adding a short SAM mid-training phase to existing OLMo-2-1B checkpoints reduced forgetting by 31% under MetaMath post-training and 40% under 4-bit quantization. The author argues that standard evaluation metrics like perplexity and MMLU may systematically favor fragile models, since they never measure stability under subsequent updates.

The Problem: Sharp vs. Flat Minima

The post opens with a scenario: two teams train 1B-parameter models with nearly identical pretraining loss and benchmark scores. After three months of post-training, one model excels on MetaMath while the other has forgotten 31% more pretrained capability. The only difference: the first team used SAM during mid-training.

The paper (arXiv:2605.02105) asks a question most of the field ignores: are pretraining minima sharp or flat?

A point on a needle tip has equally low loss and high benchmark scores, but any parameter update (post-training) knocks it off entirely.
A point at the bottom of a bowl may have slightly higher loss but tolerates large parameter changes without losing capability.

Loss Landscape Geometry

Near a critical point, loss is approximated via the Hessian:

\mathcal{L}(\theta^* + \delta) \approx \mathcal{L}(\theta^*) + \frac{1}{2}\delta^T H(\theta^*) \delta

Large Hessian eigenvalues mean a sharp minimum; small eigenvalues mean a flat one—flat minima are what makes models robust to later updates.

Three Ways to Flatten the Bowl

1. SAM (Sharpness-Aware Minimization). Instead of only minimizing loss, SAM penalizes points whose neighborhoods are steep. Its two-step update:

\tilde{\theta} = \theta_t + \rho \frac{\nabla \mathcal{L}(\theta_t)}{\|\nabla \mathcal{L}(\theta_t)\|}, \quad \theta_{t+1} = \theta_t - \eta \nabla \mathcal{L}(\tilde{\theta})

where $\rho$ controls the perturbation radius.

2. Large learning rates. Bigger steps make it easier to escape narrow local optima and reach open basins.

3. Shorter annealing periods. Rapid learning-rate decay causes the optimizer to settle prematurely in sharp optima; slower annealing allows crossing more local optima toward flatter regions.

Results

On 20M-150M parameter models across 5 downstream datasets: forgetting after post-training reduced by up to 80%.
On existing OLMo-2-1B checkpoints, adding only a short SAM mid-training phase:

| Follow-up | Forgetting reduction | |:---|:---| | MetaMath post-training | 31% | | 4-bit quantization | 40% |

Cost Concerns

SAM requires two gradient computations (~2x cost), but the OLMo-2-1B experiments show only a short mid-training SAM phase is needed—like one structural reinforcement, not reinforced bricks throughout. Larger learning rates and shorter annealing are essentially free.

The Uncomfortable Implication

The author's key argument: standard benchmarks (perplexity, MMLU, GSM8K) measure "how strong is the model now" but never "how stable is the model under subsequent updates." They may systematically select the most fragile models. This forgetting is invisible—downstream teams never see the counterfactual model that could have retained 80% more capability.

The paper does not offer a full engineering playbook. Its contribution is more fundamental: the pretraining evaluation framework itself may be wrong, and pipelines without flatness monitoring are chasing perfection on a needle tip.

Paper Details

| Item | Content | |:---|:---| | Title | Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting | | Authors | Ishaan Watts, Catherine Li, Sachin Goyal, Jacob Mitchell Springer, Aditi Raghunathan | | arXiv | 2605.02105 | | Date | May 4, 2026 | | Category | cs.LG | | Methods | SAM, large learning rate, short annealing → flat minima | | Scale | 20M-150M params, 5 downstream datasets; OLMo-2-1B validation | | Key finding | Up to 80% less post-training forgetting; 40% less quantization forgetting |