Solving Physics Olympiad Problems via Reinforcement Learning on Physics Simulators

Forum topic · 小凯 · 2026-04-15

Summary

A new arXiv paper (2604.11805) proposes using physics simulators as an alternative supervision source for training LLM physical reasoning, addressing the scarcity of internet QA data highlighted by the DeepSeek-R1 era. The authors generate random scenes in physics engines, create synthetic question-answer pairs from simulated interactions, and train LLMs with reinforcement learning. Remarkably, models trained only on synthetic simulation data show zero-shot sim-to-real transfer, improving performance on real IPhO (International Physics Olympiad) problems by 5-10 percentage points. The research areas span cs.LG, cs.AI, cs.CV, and cs.RO, with authors from CMU including Katerina Fragkiadaki and Deepak Pathak. This work suggests simulators can bypass the data bottleneck limiting LLM reasoning advances in physics.

Solving Physics Olympiad via Reinforcement Learning on Physics Simulators

This post introduces the paper "Solving Physics Olympiad via Reinforcement Learning on Physics Simulators" (arXiv:2604.11805).

Research areas: cs.LG, cs.AI, cs.CV, cs.RO
Authors: Mihir Prabhudesai, Aryan Satpathy, Yangmin Li, Zheyang Qin, Nikash Bhardwaj, Amir Zadeh, Chuan Li, Katerina Fragkiadaki, Deepak Pathak
Published: 2026-04-13

Motivation

LLM reasoning has advanced rapidly since DeepSeek-R1, but this progress has largely relied on the abundance of internet question-answer (QA) pairs. Such data is limited in scale and concentrated mainly in domains like mathematics, making it a major bottleneck going forward.

Approach

The authors show that physics simulators can serve as a powerful alternative source of supervision for training LLMs in physical reasoning:

1. Generate random scenes in physics engines 2. Create synthetic QA pairs from the simulated interactions 3. Train LLMs using reinforcement learning on this synthetic data

Results

The trained model demonstrates zero-shot sim-to-real transfer: trained purely on synthetic simulation data, it improves performance on real IPhO (International Physics Olympiad) problems by 5-10 percentage points.

---

Source: arXiv:2604.11805