IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier
Field: NLP Authors: Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan Published: 2026-09-09 arXiv: 2609.10494
Abstract
Enterprises deploy systems, not checkpoints. Usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score advertised model identifiers. The authors treat this as measurement error and give a protocol that makes it reportable. It has three parts:
1. Gold-blind capability-binding preflight — verifies that a route can execute the evaluation contract before any task reaches it. 2. Reliability-inclusive first-pass scoring rule — keeps failure in the score while keeping unsupported capability out. 3. Structurally score-blind adjudication.
The protocol (called IB2) releases its algorithms, classification tables, request contract, and manifest schemas. Its reference instantiation — 128 locked tasks and 987 assertions over document, spreadsheet, chart, tool, and database work — stays sealed: the procedure is the artifact, not the corpus.
Key Findings (across eleven systems)
- Capability availability is measurable: two complete single-route runs on identical weights later failed distinct predicates of the finalized binding gate, while a third passed that gate before a fresh run. The advertised identifier exposed neither limit.
- Discrimination is not uniform: four of seven suites saturate under a six-system band, with the spread almost entirely from governed database work and multi-tab joins; the paper reports interval-backed resolution groups, not ranks. Two of the nominal five-label output's four cuts fail multiplicity adjustment.
- Serving-arm choice matters: one declared revision and precision moved from 77.38 to 82.54 (paired interval [0.11, 10.60]), though the arms differ in access mode, harness generation, and the serving tool-call parser — and harness generation is a property of the evaluator, not any endpoint.
- Reliability inclusion changes conclusions: excluding failed responses from denominators changes the point ordering, so reliability inclusion changes a conclusion, not its wording.
*Auto-collected on 2026-09-11.*