[论文] QuoteBench: How Matched Scores Can Hide Command-Path Failures

论文概要

研究领域: ML 作者: Shangao Li, Yao Zhang, Volker Tresp, Yuanyuan Yang 发布时间: 2026-08-13 arXiv: 2608.13547

中文摘要

LLM编码智能体通过可能序列化、包装和重新解析模型输出的接口发出Bash命令。仅靠匹配执行分数无法区分命令生成错误与生成后引入的失败。QuoteBench通过在14个事件衍生家族的56个一次性任务上进行精确最终状态验证来测量这一边界，跨越生成契约与执行传输，围绕一个故意未转义的添加解析器。在插值点进行转义可以复现每个重放回复的原始路径结果，因此在披露边界下的任何恢复必须来自模型改变其生成。在八种相同窗口配置中，通过添加解析器重放相同回复将成功率降低55.4至73.2个百分点；披露为六种配置恢复了30.4至60.7个百分点，对另外两种则恢复为零或略负。原始生成在前沿几乎已饱和；边界适应仍然是区分模型的因素。GPT-5.6-sol的匹配差距为-3.6分，隐藏了-64.3分的损害和+60.7分的补偿。部署配置重新排序模型：26个可比较对中有1个反转是明确的，另有4个处于单任务边缘。命令发出智能体的评估应报告模型配置、生成契约、执行路径、操作点和最终状态验证器，而不是将匹配分数视为模型的内在属性。

原文摘要

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

--- *自动采集于 2026-08-15*

#论文 #arXiv #ML #小凯