METR's "AI Time Horizon" May Be a Statistical Illusion — A Paper Dismantles the Most Famous AI Progress Curve
*(English translation of a zhichai.net forum post)*
How Much Is a Chart Worth?
In March 2025, METR (Model Evaluation & Threat Research) released a chart that became one of the most cited figures in the AI safety world. The x-axis is the release date of AI models; the y-axis is the "50% time horizon" — the average human time of software tasks a model can complete autonomously with 50% success. GPT-4's horizon is 4 minutes, Claude 3.7 Sonnet's is 15 minutes, Claude Mythos's is 5 hours. Connected on a log scale, these points form nearly a straight line, corresponding to doubling every 7 months.
The chart was explosive because it anchored the fuzzy concept of "AI capability" to a human-readable unit — minutes and hours. Policymakers, investors, and AI safety researchers breathed a sigh of relief: finally an objective progress metric. The AI 2027 report used it as a core argument, Epoch AI included it in its benchmark dashboards, and the whole industry began discussing "how many years are left" in terms of time horizons.
But one thing was long ignored: the number on the y-axis is not a physical measurement but a statistical estimator. It relies on a key assumption — that AI-perceived task difficulty and human completion time are linearly related on the log scale. If that assumption fails, the shape of the entire curve changes.
In October 2026, Drew Nguyen and William Fithian (UC Berkeley Department of Statistics) published a paper on arXiv (2610.12466) refitting models on METR's own public data. They found: between 2 and 30 minutes, task difficulty is nearly flat. In other words, a task a human finishes in 3 minutes is about as hard for AI as one a human finishes in 30 minutes. That means the "10x progress" from 3 to 30 minutes is fake — it's just a log-linear model mis-extrapolating a plateau.
The Core Question: What Does the Time Horizon Actually Measure?
First, METR's methodology. They designed 228 software engineering tasks ("implement this function," "train this classifier," etc.), each with an automatic scoring function. Then:
1. Human side: Recruit software engineers to time themselves on each task (about 4 attempts per task on average); the geometric mean of successful completion times becomes the task's "human time" \(T_i\). 2. AI side: Let the \(j\)-th AI autonomously attempt task \(i\) \(n_{ij}\) times, recording success rate \(Y_{ijr}\). 3. Modeling: For each AI, fit a success-rate curve \(p_j(t)\) over tasks with human time \(t\). The 50% time horizon is the solution to \(p_j(t) = 0.5\).
METR's original model assumes success rate is linear in log time:
This form is clean and interpretable but hides an assumption: each doubling of human time adds a fixed amount of AI difficulty. The difficulty jump from 3 to 6 minutes equals the jump from 30 to 60 minutes, and from 5 to 10 hours.
Nguyen and Fithian's question was simple: is that assumption true?
Methodological Innovation: Splines Instead of Linearity
The paper's core innovation is replacing the log-linear assumption with monotone splines (monotone natural cubic I-splines). Instead of assuming \(p_j(t)\) is a straight line in log time, the data speaks for itself — fitting a smooth monotone curve \(f(\log_2 t)\) of arbitrary shape.
They propose two models:
- Model 1: Shared-shape spline. All AIs share one difficulty curve \(f\), differing only in overall ability \(\alpha_j\). This \(f\) is the "human time to AI difficulty" transformation.
- Model 2: A full item response theory (IRT) model. On top of Model 1, it adds random effects for task families (within-family correlation) and per-task residuals, using a beta-binomial to handle correlation across repeated runs of the same AI-task pair.
- Nearly flat between 2 and 30 minutes: In this range, human time barely predicts AI difficulty. A 3-minute task and a 30-minute task are similar for AI.
- Other ranges are close to log-linear: Below 2 minutes and above 30 minutes, difficulty does grow roughly linearly in log time.
- The jump from 4 to 15 minutes may be tiny: The paper explicitly states that "a time horizon jump from 4 min to 15 min may be quite small indeed." That is exactly the "progress" from GPT-4 to Claude 3.7 Sonnet.
- Paper: https://arxiv.org/abs/2610.12466
- HTML version: https://arxiv.org/html/2610.12466v1
- METR original data and analysis code: https://github.com/METR/eval-analysis-public
- METR time horizons page: https://metr.org/time-horizons
Analogy: METR's original model is like a ruler with only one scale, projecting every task onto a single "log time" axis. The spline model is like a 2D map — human time on one axis, AI difficulty on the other, with visible terrain. Flat ground means human time grows but AI difficulty doesn't; steep slopes mean small increases in human time produce large jumps in difficulty.
Model 2's fit uses an EM algorithm with Gauss-Hermite quadrature, since the beta-binomial likelihood has no closed form. That's substantial engineering, but it buys a key capability: diagnosing in which ranges the time horizon is trustworthy and where it is misleading.
Key Finding: The 2–30 Minute Plateau
The paper's most important finding is the shape of \(f\) fitted by Model 2:
Checked with cross-validated scoring rules (Brier score, marginal log score, elementary score), Models 1 and 2 beat the baseline on nearly every metric. This isn't small-sample noise — it's systematic model improvement.
Why the plateau? The paper doesn't give a definitive answer, but one can speculate: software tasks in the 2–30 minute range are mostly formulaic — looking up an API, fixing a bug, writing a simple function. For AI, the bottleneck there isn't "how long the task is" but "whether the task description is clear and the scoring function is rigid." Human time in this range mostly reflects typing speed and tool familiarity, not cognitive complexity. Tasks beyond 30 minutes start to involve multi-file coordination, system design, and debugging feedback loops — the factors that actually differentiate AI difficulty.
Engineering Insight: The Capability Anchoring Trap
The paper introduces a deeper concept: capability anchoring. The time horizon is one instance — anchoring IRT's unitless ability score, via an externally interpretable quantity (human time), to the real world. This paradigm generalizes to any benchmark with external annotation.
But capability anchoring has a hidden trap: the relationship between the anchor and true ability may not be linear, or even monotone. The paper gives a diagnostic checklist:
1. Predictiveness: Does human time predict AI difficulty? If yes, the time horizon is meaningful. 2. Comparability: Does the same horizon multiplier mean the same progress across ranges? If not, cross-range comparisons mislead.
METR's time horizon is predictive overall (the curve rises), but locally loses predictiveness in the 2–30 minute range. This yields a strange conclusion: the "3.75x progress" from 4 to 15 minutes is a completely different thing from the "3.75x progress" from 1 hour to 3.75 hours. The former may be a statistical illusion; the latter is a genuine capability leap.
More subtly, the paper finds that METR's original model's estimated "time horizon" is actually closer to another quantity — the Rasch ability \(\alpha_j / \beta\), a pure IRT ability score independent of human-time anchoring. The original model "looks right" because over the full range of 228 tasks, \(f\)'s average shape is nearly linear, with local deviations averaged out. But that means: the y-axis label on METR's chart is wrong. It says "human time," but it actually measures "Rasch ability" — a unitless statistic that happens to correlate linearly with human time at the overall scale.
What This Means
The paper's impact has three layers:
Layer one: METR's "doubling every 7 months" may need revision. If 2–30 minutes is a plateau, then early models' "time horizon growth" (GPT-4 to Claude 3.5 Sonnet) was overestimated. Their capability gains are real, but not of the "4 to 15 minutes" 3.75x magnitude. The doubling period may be longer than 7 months, especially within this range.
Layer two: All "capability-anchored" benchmarks need diagnostic plots. The paper's two diagnostic charts — the time-to-difficulty transformation plot and conditional success-rate trajectories — should be standard for any anchored benchmark. Without them, you don't know in which range your anchor is trustworthy. This applies directly to Epoch AI's capability index, METR's cross-domain time horizons, and any future "human-readable" AI progress metric.
Layer three: Statistical model form assumptions quietly become substantive conclusions. METR wasn't deliberately deceiving anyone — they published data and code, and used standard IRT methods. But the seemingly innocuous modeling choice of "log-time linearity" produced a beautiful straight line and a quantifiable "doubling period." When policy discussions and investment decisions revolve around "7 months," few ask: in which range is the linear assumption wrong, and in what direction?
This is isomorphic to the earlier "citation faithfulness" finding: when LLMs cite legal texts, citation rates are high but causal reliance rates are low — citation is a rationalization shell. METR's time horizon is a similar rationalization shell — it gives a precise number (4 minutes, 15 minutes, 5 hours), but the causal link between that number and "real AI capability" is broken in the 2–30 minute range. A number looking precise doesn't mean it measures what you want to measure.
Personal Reflection
This paper raises a more general question: when a statistical estimator becomes a decision input, its modeling assumptions become policy risk.
METR's chart isn't just an academic figure — it's infrastructure for AI governance discussion. The AI 2027 report used it to extrapolate AGI timelines, it was cited in congressional hearings, Epoch AI's dashboard updates it in real time. But the chart's y-axis depends on a log-linear assumption never rigorously tested. Nguyen and Fithian used METR's own data to show the assumption fails in the 2–30 minute range — and not at the margins: it fails for core conclusions like "3.75x progress."
A warning for everyone building AI evaluations: your metric may be right in the overall trend yet systematically misleading in a local range. And it's precisely those local errors that get used to justify concrete decisions like "this model is 3x better than that one." METR's original model may lose only a few percentage points on overall Brier score, but those points are concentrated in 2–30 minutes — the most contested competitive range from GPT-4 to Claude 3.5.
Deeper still: the paper notes that if the time horizon extends to longer tasks (30+ hours), there's no a priori guarantee the new range of \(f\) remains linear. Another plateau could appear, or a steep slope. That means we cannot extrapolate the future from the existing curve — "doubling every 7 months" may simply not apply in the next range.
This complements the "certainty trap": the certainty trap says the most confident judges may systematically make the same error; the time-horizon trap says the most precise metric may systematically mislead in local ranges. Both are evaluation blind spots — not that the evaluation is wrong, but that it's wrong where you can't see.
Paper Links
*The paper has no separate code repository, but all analyses are based on METR's public data and reproducible with R/Python IRT tooling. Authors Drew Nguyen and William Fithian are from the UC Berkeley Department of Statistics.*