QuoteBench: How Matched Scores Can Hide Command-Path Failures
Field: Machine Learning
Authors: Shangao Li, Yao Zhang, Volker Tresp, Yuanyuan Yang
Date: 2026-08-13
arXiv: 2608.13547
Summary
LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation.
Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two.
Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins.
Key Findings
- Matched scores are misleading. Surface-level execution-match metrics can hide large underlying command-path failures introduced by serialization, wrapping, and reparsing.
- QuoteBench isolates the boundary. It uses 56 one-shot tasks from 14 incident-derived families and exact final-state validation, crossing the generation contract with the execution transport around one deliberately unescaped added parser.
- Replay damage is large. Running the same reply through the added parser cuts success by 55.4–73.2 percentage points across eight same-window configurations.
- Disclosure recovery varies. Telling the model about the boundary recovers 30.4–60.7 points for six configurations, but zero or slightly negative for the remaining two — adaptation, not raw generation, differentiates frontier models.
- A concrete example. GPT-5.6-sol shows only a -3.6-point matched gap, which conceals -64.3 points of damage and +60.7 points of compensation.
- Rankings depend on deployment. Among 26 comparable model pairs, one ranking reversal is unambiguous and four more sit on single-task margins, so deployment configuration changes leaderboards.
- Evaluation recommendations. Command-issuing agent evaluations should report the model configuration, generation contract, execution path, operating point, and final-state validator — not treat a matched score as an intrinsic model property.
Original Abstract
> LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.
---
*Auto-collected 2026-08-15*