IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier (arXiv:2609.10494)

Forum topic · 小凯 · 2026-09-11

Summary

This paper introduces IB2, a protocol for auditing enterprise AI systems based on their actual serving route rather than the advertised model identifier. The authors argue that usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score only advertised model identifiers. IB2 has three parts: a gold-blind capability-binding preflight that verifies a route can execute the evaluation contract before tasks reach it; a reliability-inclusive first-pass scoring rule that keeps failures in scores while excluding unsupported capabilities; and structurally score-blind adjudication. The reference instantiation includes 128 locked tasks and 987 assertions over document, spreadsheet, chart, tool, and database work. Across eleven systems, the authors find capability availability is measurable (two identical-weight runs failed distinct binding predicates), discrimination is uneven (four of seven suites saturate under a six-system band), serving-arm choice moved one declared revision and precision from 77.38 to 82.54, and reliability inclusion changes scoring conclusions, not just wording.

Overview

Field: NLP Authors: Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan Published: 2026-09-09 arXiv: 2609.10494

Abstract (full translation)

Enterprises deploy systems, not checkpoints. Usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score advertised model identifiers. We treat this as measurement error and give a protocol that makes it reportable. It has three parts. A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it; a reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out; and adjudication is structurally score-blind. We call the protocol IB2 and release its algorithms, classification tables, request contract, and manifest schemas. Its reference instantiation, 128 locked tasks and 987 assertions over document, spreadsheet, chart, tool and database work, stays sealed: the procedure is the artifact, not the corpus.

Key findings (eleven systems)

Capability availability is measurable: two complete single-route runs on identical weights later failed distinct predicates of the finalized binding gate, while a third passed that gate before a fresh run. The advertised identifier exposed neither limit.
Discrimination is not uniform: four of seven suites saturate under a six-system band, with the spread almost entirely from governed database work and multi-tab joins, so the authors report interval-backed resolution groups, not ranks; two of the nominal five-label output's four cuts fail multiplicity adjustment.
Serving-arm choice matters: one declared revision and precision moved from 77.38 to 82.54 (paired interval [0.11, 10.60]), though the arms differ in access mode, harness generation, and the serving tool-call parser; harness generation is a property of the evaluator, not any endpoint.
Reliability inclusion changes conclusions: excluding failed responses from denominators changes the point ordering, so reliability inclusion changes a conclusion, not its wording.

Protocol components

1. Gold-blind capability-binding preflight — verifies a route can execute the evaluation contract before any task reaches it. 2. Reliability-inclusive first-pass scoring — keeps failure in the score while keeping unsupported capability out. 3. Structurally score-blind adjudication — adjudicators cannot see scores.

The algorithms, classification tables, request contract, and manifest schemas are released; the 128-task / 987-assertion corpus remains sealed — the procedure is the artifact, not the corpus.

---

*Auto-collected on 2026-09-11*