English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original
智柴网 中文原文

Beyond the Board: How Pretraining Loss Predicts RL Gains in Reasoning Models

Forum topic · 小凯 · 2026-07-20

Summary

A 2026 paper by Shen, Li, Rahman, et al. (arXiv:2607.16097) investigates how reasoning ability emerges from pretraining to reinforcement learning using a controlled chess testbed. Training transformer models from 5M to 1B parameters on human chess games, the authors discover a near-linear relationship between pretraining loss and final post-RL performance, enabling prediction of RL outcomes without running RL. They also show that RL does not merely sharpen existing SFT strategies; for hard puzzles where SFT assigns near-zero probability to correct moves, RL systematically discovers novel solutions. The same pretraining-RL scaling pattern transfers to a 1B-parameter math model, suggesting a domain-general principle. Practical implications include using pretraining loss as a cheap proxy for RL potential, allocating more compute to pretraining, and designing RL reward functions that allow non-human reasoning paths. The work reinforces the trend seen in DeepSeek-R1 and OpenAI o1-style reasoning models.

Key Points

  • Controlled testbed: The authors train transformer chess models from 5M to 1B parameters across pretraining data scales of 10M to 1B tokens, then apply SFT and RL on chess puzzles to isolate how pretraining shapes RL gains.
  • Pretraining loss predicts RL performance: A near-linear mapping RL_final ≈ a × pretraining_loss + b holds across model sizes and data scales. Higher pretraining token counts also produce steeper RL learning curves, meaning better pretrained models learn faster under RL, not just higher.
  • RL discovers new strategies, not just sharpens old ones: On easy puzzles, RL amplifies correct moves SFT already prefers. On hard puzzles where SFT assigns near-zero probability to the correct move, RL causes the model to systematically select those moves. This contradicts the "sharpening" intuition and mirrors AlphaZero's non-human chess discoveries.
  • Cross-domain validation: The same scaling pattern reappears in a 1B-parameter math model trained on math text, SFT, and RL with verifiable rewards, suggesting a domain-general law linking pretraining quality to RL outcomes.
  • Practical implications for LLM training:
  • Pretraining sets the ceiling; RL can rearrange the layout but cannot raise the foundation. Compute invested in RL cannot fix undercooked pretraining.
  • Pretraining loss is an underused proxy for downstream reasoning potential. Use it for checkpoint selection, early-stopping, and compute budgeting rather than waiting for full RL runs.
  • RL reward design matters: rewarding only final correctness (not human-style reasoning format) is what enables discovery of novel solutions.
  • The result motivates joint cross-stage optimization, where pretraining objectives are designed to predictively maximize post-RL performance.
  • Methodological argument: Chess is proposed as a reusable "AI laboratory" because its rules are deterministic, its state space is large yet verifiable, and it has millennia of human knowledge for comparison.

Source Reference

Shen, J., Li, A., Rahman, S., et al. (2026). *Understanding Reasoning from Pretraining to Post-Training*. arXiv:2607.16097. https://arxiv.org/abs/2607.16097

Tags

#arxiv#pretraining#reinforcement-learning#reasoning#llm#scaling-laws#chess#testbed

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178446963