Summary
SciExam for ENSO is a benchmark that evaluates whether language-model agents can perform genuine open-ended scientific research by building low-order stochastic models of the El Nino-Southern Oscillation (ENSO), the dominant mode of interannual climate variability, from real observations. Within a six-hour budget, agents process observational data, write their own diagnostic tools (which are then frozen), and develop a model using only these diagnostics as feedback. Hidden graders test whether the resulting model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, applying identical scoring to published models for comparison. Across 12 agent systems, 6 produced models scoring above published models, mainly through better reconstruction and prediction. Simplified forms of stronger models each aligned with one of two competing explanations for ENSO's warm-cold asymmetry, an open scientific debate never mentioned in the task. Controlled runs under varied information conditions show the scores do not come from memorizing outdated observational records, and the information received shapes how the system models. The benchmark enables assessment of agent research without known answers.
Paper Overview
Research Area: ML
Authors: Yinling Zhang, Langchen Liu, Dongbin Xiu, Xueyan Zou, Xu Kuang, Mengdi Wang, Shilong Liu
Published: 2026-10-07
arXiv: 2610.10513
Summary
Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer — none of which can tell whether a new scientific model is valid. SciExam for ENSO is a benchmark in which agents build low-order stochastic models of ENSO (El Nino-Southern Oscillation), the dominant mode of interannual climate variability, from real observations.
Within a six-hour budget, agents process the observations, write their own diagnostics (which are then frozen), and develop a model using only these diagnostics as feedback. Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, scoring published models in the same way for comparison.
Key Findings
- Of 12 agent systems evaluated, 6 produced models that scored above the published model, mainly through better reconstruction and prediction.
- Simplified forms of the stronger models were each compatible with one of two competing explanations for ENSO's warm-cold asymmetry — an open scientific debate that was never mentioned in the task.
- Controlled runs under varied information conditions showed the strongest system's scores were not derived from memorization of outdated observational records, and the information it received shaped its modeling approach.
SciExam for ENSO therefore enables evaluation of agent-driven research without requiring known answers, and demonstrates that agents can already build competitive scientific models whose structure engages questions scientists are still actively debating.
---
*Auto-collected on 2026-10-09*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635760