When Perfect Scores Hide Failures: QuoteBench and the Evaluation Blind Spot in Command Paths

Forum topic · ✨步子哥 · 2026-08-14

Summary

QuoteBench exposes a blind spot in LLM benchmarking: reported success rates are not intrinsic model properties but products of four variables—model configuration, generation contract, execution transport, and final-state validator. Using a 2×2 crossed design (raw vs. disclosed contract × raw vs. nested transport), QuoteBench decomposes the "matched gap" between matched configurations into transport damage and contract-conditioned compensation via an exact accounting identity. Results are striking: GPT-5.6-sol shows only a −3.6 matched gap, but this masks −64.3 transport damage cancelled by +60.7 compensation; Qwen3.5-27B compensates zero (−55.4); Gemini-3.1-Flash-Lite gets worse when informed (−64.3). Deployments replayed via real ssh localhost commands reproduce nested damage exactly, and naive JSON embedding causes similar 51.8–66.1 point losses, showing the mechanism is generic unescaped transformation, not shell-specific. Key engineering takeaways: report contract and transport with scores, escape at interpolation points (correct escaping restores all 448 test pairs with zero loss), or write model output to temp scripts. Deployment configuration can even reorder model rankings.

Key points

QuoteBench argues that LLM benchmark scores are often artifacts of the evaluation pipeline itself. A model's command may be correct, yet fail after passing through a shell-nesting layer (e.g., ssh localhost "R") that strips a level of double quotes.

Scores are not model properties. Success rate is a function of four variables: (1) model configuration, (2) generation contract, (3) execution transport, (4) final-state validator. Standard benchmarks report a blended product of all four.
2×2 crossed design: raw contract (model outputs a Bash program, executed directly) vs. disclosed contract (model is told its reply R will be interpolated into bash -c "R"), crossed with raw transport vs. nested transport (one extra layer of double-quote parsing).
Exact decomposition:

Y_{NN} - Y_{RR} = (Y_{RN} - Y_{RR}) + (Y_{NN} - Y_{RN})

Matched gap = transport damage + contract-conditioned compensation. Large damage can be cancelled by large compensation, making the matched score look fine.

Selected results (Table 4)

| Model | R·R | R·N | N·R | N·N | Damage | Compensation | Matched gap | |---|---|---|---|---|---|---|---| | GPT-5.6-sol | 94.6 | 30.4 | 55.4 | 91.1 | -64.3 | +60.7 | -3.6 | | GPT-5.5 | 100.0 | 28.6 | 50.0 | 89.3 | -71.4 | +60.7 | -10.7 | | Gemini-3.5-Flash | 96.4 | 28.6 | 67.9 | 58.9 | -67.9 | +30.4 | -37.5 | | Qwen3.5-27B | 85.7 | 30.4 | 83.9 | 30.4 | -55.4 | 0.0 | -55.4 | | Gemini-3.1-Flash-Lite | 78.6 | 19.6 | 80.4 | 14.3 | -58.9 | -5.4 | -64.3 |

GPT-5.6-sol's near-zero matched gap is the result of −64.3 damage cancelled by +60.7 compensation. Qwen3.5-27B cannot compensate at all; Gemini-3.1-Flash-Lite gets worse when told about the boundary.

Practical implications

> The command interface is part of the evaluated system, not neutral plumbing.

1. Report contract and transport with every score, not a single blended success rate. 2. Escape at the interpolation point. Correct escaping restores all 448 public test pairs to raw-path results — zero loss. 3. Or use a temp script: write the model output to a file to preserve program boundaries.

Rankings can reorder under deployment configuration: selecting by raw success picks GPT-5.5 (56/56 raw, 50/56 nested), while the path-aware pick reaches 51/56.
Not shell-specific: naive JSON string embedding causes 51.8–66.1 point losses — same mechanism of "unescaped transformation." Any serialization boundary (JSON tool calls, YAML, XML templates) is vulnerable; correct round-trip serializers have zero loss.
Ecologically valid: replaying stored raw replies through a real ssh localhost "R" command reproduces the synthetic nested damage to the decimal in seven of eight configurations.

Positioning

QuoteBench extends a family of "evaluation blind spot" findings: you optimize what you measure, and what you don't measure is where problems hide — here, the benchmark pipeline itself, not just the task selection.

Resources

Code: https://github.com/LeonardNJU/quoteBench
Paper (arXiv): https://arxiv.org/abs/2608.13547
HTML full text: https://arxiv.org/html/2608.13547v1

One-line takeaway: matched score = damage + compensation. When both are large with opposite signs, the "normal" score you see is two failures cancelling each other out. QuoteBench takes that illusion apart.