When Perfect Scores Hide Failures: QuoteBench and the Blind Spot of Command-Path Evaluation

Key points

• Core claim: A benchmark score like "GPT-5.5 scores 56/56 on Bash" is not a model property — it is a function of four variables: model configuration, generation contract, execution transport, and final-state validator. Standard evaluation reports only a mixed product of all four and reads it as "model capability."
• The motivating example: A model outputs mv "old file.txt" "new file.txt", which works locally. But when executed via ssh localhost "R", one layer of quotes is consumed by nested shell parsing, splitting the filename into two arguments. The model didn't fail — the pipeline did — yet the benchmark still records local-path success.
• 2×2 crossover design:

| | Raw transport (R) | Nested transport (N) | |---|---|---| | Raw contract (R) | R·R (direct output, direct execution) | R·N (direct output, nested parsing) | | Disclosed contract (N) | N·R (model told the boundary, direct execution) | N·N (model told the boundary, nested parsing) |

R·R and N·N are the "matched scores" standard benchmarks report; R·N and N·R are diagnostic cells isolating pipeline contributions.

• Exact decomposition identity:
\[Y_{NN} - Y_{RR} = (Y_{RN} - Y_{RR}) + (Y_{NN} - Y_{RN})\]

Matched gap = transport damage + contract-conditioned compensation. This is an accounting identity, not an approximation. When damage and compensation are both large with opposite signs, the matched score can look near zero while hiding two large opposing effects.

• Headline data (paper Table 4):

| Model | R·R | R·N | N·R | N·N | Damage | Compensation | Matched gap | |---|---|---|---|---|---|---|---| | GPT-5.6-sol | 94.6 | 30.4 | 55.4 | 91.1 | -64.3 | +60.7 | -3.6 | | GPT-5.5 | 100.0 | 28.6 | 50.0 | 89.3 | -71.4 | +60.7 | -10.7 | | Gemini-3.5-Flash | 96.4 | 28.6 | 67.9 | 58.9 | -67.9 | +30.4 | -37.5 | | Qwen3.5-27B | 85.7 | 30.4 | 83.9 | 30.4 | -55.4 | 0.0 | -55.4 | | Gemini-3.1-Flash-Lite | 78.6 | 19.6 | 80.4 | 14.3 | -58.9 | -5.4 | -64.3 |

Two models with identical matched scores may be wildly different systems: one walking a tightrope (large damage, large compensation), one strolling on flat ground (small damage, small compensation).

• Medical analogy: A blood-pressure reading is a matched score — the end state after all compensatory mechanisms act. An ECG is the crossover decomposition — it reveals each mechanism's individual contribution. QuoteBench does not invent a new metric; it splits a blended number into its components.
• Key engineering insight: *"The command interface is part of the evaluated system, not neutral plumbing."* The shell, wrapper, and parser between model and execution environment change the very commands being evaluated.
• Three practical recommendations:
1. Report contract and transport: not "95% on Bash" but "95% under raw contract + raw transport, 72% under disclosed contract + nested transport." 2. Escape at the interpolation point: if interpolating model output into bash -c "R", escape correctly — the paper verifies this restores all 448 public test pairs to raw-path results, zero loss. 3. Or use a temp script: write the model output to a file and execute it, preserving program boundaries; the cost is one file lifecycle, and it eliminates all quoting issues.
• Deployment re-ranks models: selecting by raw success picks GPT-5.5 (56/56 raw), which reaches 50/56 on the nested path, whereas the path-aware pick reaches 51/56. Ignoring the path changes which model wins — this is not a small gap, it is choosing the wrong model.
• Beyond shell: the paper's discussion extends the decomposition to JSON boundaries — naive JSON string embedding re-parsed loses 51.8–66.1 points, the same magnitude as nested shell transport. The mechanism is "unescaped transformation," not shell-specific. Any pipeline inserting serialization/deserialization (JSON tool-calls, YAML configs, XML templates) can introduce the same damage; correct round-trip serializers lose nothing.
• Real-world validity: replaying each stored raw reply through a real ssh localhost "R" remote command reproduces the nested damage to the decimal for seven of eight configurations and within one task for the eighth. SSH, CI/CD, container exec, and remote wrappers all fall within this evaluation scope.
• Conceptual lineage: the post situates QuoteBench in a series of "evaluation blind spot" findings (Epanorthosis, TokenBudget, QuantiBias, Progressive Cramming, Looping Is Not Reliability): *you optimize what you measure, and unmeasured things are where problems hide*. QuoteBench extends this from "which tasks you test" to "which pipeline you test through."

Author's reflection

There is no such thing as "a model's capability on a task" — only the result of a five-tuple: model × contract × transport × validator × operating point. Compressing this into one number discards exactly the information that matters most at deployment: the internal interplay of damage and compensation. Optimization granularity should match the granularity of the thing being optimized; using a blended score for model selection is like diagnosing a local infection from average body temperature. The pipeline is not a supporting actor — it is part of the script.

Links

• Code: https://github.com/LeonardNJU/quoteBench
• arXiv: https://arxiv.org/abs/2608.13547
• HTML full text: https://arxiv.org/html/2608.13547v1

One-line takeaway: matched score = damage + compensation. When both are large with opposite signs, the "normal" you observe is the illusion of two failures cancelling each other. QuoteBench takes that illusion apart.