Grouping the Stochastic Machine: Why Precision, Not Capability, Is the New Frontier Differentiator for LLMs

Forum topic · 小凯 · 2026-08-21

Summary

An arXiv paper (2608.19140) by George Andrikopoulos argues that frontier language models should be evaluated on precision rather than capability. Borrowing the marksman's distinction—capability is where the average shot lands, precision is the size of the group—the author claims models have saturated accuracy, so what now differentiates systems is how tightly outputs concentrate around the target across repeated identical requests. The paper makes three claims: benchmark culture systematically reports central tendency instead of spread; precision can be measured cheaply and without circularity by running deterministic scored tasks multiple times at fixed temperature and computing per-task consistency; and the measurement is decision-guiding, distinguishing consistent failures (tight groups, fixable via operational discipline) from scattered failures (wide groups, requiring model or sampling changes). A grouping metric and tooling are defined, illustrated by a field study where a measured gap was fully closed by a single rule (0/5 to 5/5), while tasks written from the rule itself found no value—showing the discipline's value lies in measurement on real work.

Overview

Field: Machine Learning Author: George Andrikopoulos Published: 2026-08-19 arXiv: 2608.19140

Frontier language models are compared, marketed, and benchmarked on capability — what their best or average output can achieve. The author argues this measures the wrong axis. The models have saturated accuracy: their mean output lands on the target. What now separates one system from another in practice is precision: how tightly concentrated their outputs are around that target across repeated, identical requests.

Borrowing the marksman's distinction, capability is where the average shot lands; reliability is the size of the group.

Three Claims

1. Precision, not capability, is the frontier differentiator between systems, and benchmark culture systematically fails to measure it, reporting central tendency rather than spread. 2. Precision is measurable cheaply and without circularity — run a fixed suite of deterministically scored tasks multiple times at fixed temperature and compute consistency of per-task outcomes. No model-in-the-loop scorer is needed. 3. The measurement is decision-guiding, not merely descriptive: it distinguishes consistent failures (tight groups off-center, correctable through operational discipline — aim adjustment) from scattered failures (wide groups, only correctable by changing the model or its sampling — the rifle problem).

Contributions

Defines a grouping metric and specifies a tooling implementation.
Shows that tracking grouping of human-AI pairs over time yields a compound signal needed for field studies.

First Real-World Run

The first field run (since replicated) illustrates both the method and its key limitation: a measured gap was fully closed by a single rule (0/5 → 5/5), while a suite of tasks written from the rule itself found no value — because frontier models already embody the explicit good practice. This establishes that the discipline's value is discovered through measurement on real work, not by constructing benchmarks from its own rulebook.

--- *Auto-collected on 2026-08-21*