Summary
This paper by Drew T. Nguyen and William Fithian (arXiv:2610.12466) re-examines METR's 50% time-horizon metric, which expresses AI capability as the human completion time of software tasks an AI solves with 50% probability. Using 228 tasks and 26 AI systems, the authors replace the standard assumption that AI task difficulty depends linearly on the log of human time with spline and item-response-theory models. The fitted spline acts as a function converting human time into AI difficulty: it is nearly flat between 2 and 30 minutes but close to linear elsewhere. A key implication is that jumping from 3 minutes to 30 minutes is much easier for AI than jumping from 30 minutes to 5 hours, even though both are a 10x multiplier. The paper also contributes improved time-horizon point estimates that outperform prior methods under a cross-validated suite of strictly proper scoring rules, plus diagnostic plots for assessing construct validity. The authors recommend interpreting time horizons alongside these diagnostics, especially when new benchmarks are proposed or existing ones extend to longer tasks.
Paper Overview
- Field: Machine Learning
- Authors: Drew T. Nguyen, William Fithian
- Published: 2026-10-08
- arXiv: 2610.12466
Summary
METR's 50% time horizon measures the human completion time of software tasks that an AI solves with 50% probability, allowing AI capabilities to be expressed in interpretable units.
On 228 tasks and 26 AI systems, the authors recompute the time horizons using splines and item-response theory, relaxing the assumption that the AI difficulty of a task depends linearly on the log of human time. The fitted spline can be interpreted as a function that *converts* human time to AI difficulty: it is nearly flat in a region from 2–30 minutes but close to linear elsewhere. Hence, a time-horizon jump from 3 minutes to 30 minutes is much easier than one from 30 minutes to 5 hours, despite the same 10x multiplier.
Contributions
- Time-horizon point estimates that perform better under a cross-validated suite of strictly proper scoring rules.
- Diagnostic plots for assessing the construct validity of the time-horizon metric.
- A recommendation to interpret time horizons together with the diagnostic plots, especially when new benchmarks are proposed or existing benchmarks are extended to longer tasks.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635791