Statistical Audit of METR's AI Time-Horizon Curve: The Scale Isn't Uniform

Forum topic · 小凯 · 2026-10-09

Summary

A detailed Chinese-language explainer reviews a 2026 statistical paper (arXiv:2610.12466) by UC Berkeley's Drew T. Nguyen and William Fithian examining METR's influential 'time horizon' metric — the measure behind the widely cited exponential curve of AI capability growth, referenced by The New York Times, Bengio, and others. The paper replaces METR's assumption that task difficulty grows linearly in log2(human time) with a shared monotonic spline and an explanatory IRT model, which beat the baseline on all 12 proper scoring rule metrics under task-family cross-validation. Key finding: in the 2–30 minute range, human time barely predicts AI difficulty — a 'doubling' of time horizon there means far less capability gain than jumps in steeper regions. The analysis also finds run-to-run correlation (83% of runs agree on outcome vs ~60% expected under independence), requiring beta-binomial modeling. Crucially, the authors do not dispute the exponential trend itself — release dates never entered the model — only the metric's scale calibration.

The Curve That Directs the World — and Its Scale

*Part one of the "Recursive Dilemma of Evaluation" trilogy: the capability scoreboard.*

Two Rulers That Measure Badly

The post opens with two analogies: a bathroom scale whose needle barely moves in one range and swings wildly in another, and a thermometer with an insensitive middle zone. Numbers from instruments feel like reality, but measurement is a *translation* of reality — and every translation has its own distortions.

The subject is a curve that, over 18 months, appeared in The New York Times, was compared to Moore's Law and the Keeling curve, and was cited by Turing laureate Yoshua Bengio and Bernie Sanders to argue for slowing AI development.

The Curve Everyone Has Seen

METR, a Berkeley non-profit dubbed "the world's informal AI judge," measures models on a metric of its own creation: the time horizon — the human-time-length of tasks an AI can complete with 50% probability. METR compiled 228 software engineering tasks across 3 suites and 79 task families, estimated human completion times (geometric mean of ~4 successful human attempts per task; 29% of tasks expert-estimated), then ran each AI multiple times.

Plotting 26 models' time horizons by release date yields a straight line on log axes — exponential growth. The data itself, notably, never uses release dates; the exponential shape emerges from task outcomes alone.

The Problem on the Mirror's Back Side

Unlike transistor counts or CO₂ concentrations (direct physical measurements), the time horizon's y-axis is a statistical estimate built on model assumptions. METR's original logistic regression assumes task difficulty is *linear* in log₂(human time) — i.e., every doubling of human time adds the same amount of difficulty.

Nguyen and Fithian ask: is this line actually straight?

Two New Tools

Model 1 replaces linearity with a shared monotonic spline (4 internal knots) for the human-time → difficulty mapping. All 26 AIs share one function shape *f*, making the conversion table itself inspectable.
Model 2 is an explanatory IRT model: each task's difficulty θᵢ = f(log₂Tᵢ) + family effect ξ + individual deviation ε.

Testing used 12 proper scoring rules (log score, Brier score, elementary binary scores at q=0.5 and q=0.8, each with three task-weighting schemes) plus 5-fold cross-validation grouped by task family. The baseline was Barry (2026)'s shared-slope logistic model — itself already better than METR's original.

Result: the new models beat the baseline on all 12 metrics. METR's own earlier spline attempts (Kwa 2026) failed because they fit separate splines per AI; sharing one shape surfaced the signal.

The Flat Twenty Minutes

Overall, log human time has good signal-to-noise — the ruler is mostly straight. But in the 2–30 minute range, the spline is nearly flat: doubling human time there barely moves AI difficulty.

So a "10× jump" from 3 to 30 minutes happens almost entirely in the flat zone — limited real capability gain — while a 10× jump from 30 minutes to 5 hours climbs a steep slope, where each minute of human time genuinely converts to difficulty. Same multiplier, very different substance. Headline claims like "time horizon doubled" must be read against where on the ruler they occur.

How to Read the Chart Now

1. Direction of growth: trustworthy. No release dates were used; the exponential form is genuine. 2. Size of steps: range-dependent. In 2–30 minutes, time-horizon movement weakly indicates capability; outside, doublings matter again. 3. Single numbers < whole curve. Each time horizon is a point estimate at a 50% threshold with three noise layers (family effects, task heterogeneity, run correlation).

The Lie of 83 Coins

METR treated repeated runs as independent coin flips. If independent, all runs agreeing on a task should occur ~60% of the time; in the data it occurs 83% — same AI, same task, runs rise and fall together (environment, dependencies, prompt noise). Model 2 handles this with a beta-binomial overdispersion parameter ρ. Inflated independence overstates the information per task, weakening confidence in the 50% threshold and everything built on it.

Twelve Checks, One Verdict

The authors report twelve scoring metrics × three weightings plus family-grouped cross-validation — validating the instrument from twelve windows onto the same mountain.

A Non-Uniform Scale Is Not a Fake Curve

The paper does not claim the METR curve is fraudulent: dates never entered the model, the ruler is mostly straight, and the improvements rest on firmer statistical ground. What changes is the *reading*, not the direction. "Doubling" means different things at different points of the scale.

The post situates this within a broader ecosystem of METR critiques (shash42 2025, Witkin 2026, Gregory Lewis 2026, Moss 2026, Kwa & Cheng 2025 on construct validity, Epoch's IRT-based Capabilities Index), and credits METR for releasing all task, model, and run data publicly — open measurement instruments deserve open audits.

The trilogy's thesis: once scoreboard numbers enter boardrooms, hearings, and budgets, their units are no longer technical details but public facts. Read the ruler before you read the number.

References

1. Nguyen, D. T., & Fithian, W. (2026). *On the estimation and validity of AI time horizons—a statistical look at the METR plot*. arXiv:2610.12466. 2. Barry, J. (2026). Shared-slope logistic model for METR time horizons. METR research notes. 3. Kwa, J. (2026). Spline approaches to time horizon estimation. METR research notes. 4. Kwa, J., & Cheng, K. (2025). Construct validity of time-horizon metrics beyond software tasks. 5. Mertens, et al. (2026). On the generalizability of capability horizon constructs. 6. Moss, R. (2026). Statistical critiques of AI capability benchmarks. 7. shash42 (2025). Criticisms of METR's time horizon methodology. LessWrong / Substack. 8. Witkin, A. (2026). Commentary on METR evaluation claims. EA Forum. 9. Gregory Lewis (2026). Notes on capability measurement. LessWrong. 10. Ho, et al. (2025). Epoch Capabilities Index.