FrontierPhysics: An End-to-End Benchmark Evaluating AI Agents for Physics Research

Forum topic · 小凯 · 2026-11-05

Summary

FrontierPhysics is a new end-to-end benchmark for evaluating AI agents on full physics research workflows, introduced by a large team of 42+ physicists and published on arXiv (2611.06910). Unlike existing physics benchmarks that test fixed questions, isolated sub-tasks, or paper reproduction, FrontierPhysics assesses both research quality and correctness. It contains 56 tasks built by 42 physicists based on their own research, spanning nine domains across theoretical, computational, and experimental physics. Each task requires an agent to review literature, formulate a research plan, conduct research and experiments, and produce structured outputs plus a scientific manuscript. Evaluation combines deterministic verifiers with expert-authored rubrics to judge whether agent research is correct and publishable. Across 22 models tested in 26 configurations, no configuration passed more than 46 of its 168 runs. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.

Overview

Field: Machine Learning Published: 2026-11-04 arXiv: 2611.06910

Key Contributions

FrontierPhysics benchmark: an end-to-end benchmark with 56 tasks created by 42 physicists based on their own research.
Broad coverage: nine domains spanning theoretical, computational, and experimental physics.
Full research pipeline: each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts.
Hybrid evaluation: combines deterministic verifiers with expert-authored rubrics to assess whether agent research is both correct and publishable.

Results

Evaluated 22 models across 26 configurations; no configuration passed more than 46 of its 168 runs.
Manual review of agent trajectories indicates that frontier agents' primary weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than physics knowledge gaps or execution errors.

Abstract (original)

AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations... *(truncated in source)*