[论文] Higher-Scoring Coding Models Did Not Get More Work Done
论文概要
研究领域: ML
作者: Yining Hua
发布时间: 2026-11-17
arXiv: 2611.23216
中文摘要
新编程模型以编码基准分数提升来宣传,但尚无研究验证高分模型是否完成更多实际工作。2026 年 6 月 12 日美国出口管制禁止访问 Claude Fable 5 共 19 天,迫使用户切换。多数原 Fable 5 用户转用 Opus 4.8(SWE-Bench Pro 低 11 分)。我们将此视为自然实验——人和项目不变,仅模型变化。追踪 297 名 Fable 5 用户在前中后三期的 GitHub PR,以 8,908 名一直用 Opus 4.8 的开发者为对照。双重差分模型显示,Fable 5 使 PR 七日合并率仅提升 1.4 个百分点 [-1.4; +4.3],不显著。五次模型升级均无显著提升(加权均值 -0.9 pp),所有估计均为负。每周合并 PR 数亦无显著增加(+2.4%)。
原文摘要
New coding models advertise themselves on score improvements of coding benchmarks. Yet no study has tested if a model that scores higher on these benchmarks gets more real work done. On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days, which forced users to switch to other models. During this time, most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5). We treat this suspension as a natural experiment, where the people and projects stayed the same and only the model changed, so we could isolate the effect of Fable 5 on user productivity. We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the cont...
自动采集于 2026-11-19
#论文 #arXiv #ML #小凯