[论文] [论文] SWE-Serve: Benchmarking Agentic Engineering For Production I...

论文概要

研究领域: Agent 作者: Jennifer Williams, Dave Farris, Jeff Farris, Jiantao Jiao 发布时间: 2026-09-22 arXiv: 2609.26777

中文摘要

我们推出 SWE-Serve——评估智能体生产推理工程能力的基准。实现一个推理功能常需在推理栈上协调多处改动：模型支持、运行时执行与公开 API。现有基准覆盖有限：仓库级 SWE 基准不针对推理；通用终端智能体基准仅含少量推理任务；专门推理基准又多关注孤立内核生成或性能优化，而非仓库级生产特性实现。SWE-Serve 提供 53 个扎根于 SGLang 近期生产改动的仓库级任务，覆盖六个推理工程家族；每个任务在 CPU 或单 GPU（H100）上执行，以隐藏功能与回归测试评估，含端到端服务测试与校准性能门槛；可执行空操作与预言机对照、对抗验证者审查、闭卷执行保障评测完整性。11 个模型 × 31 个模型-effort 配置中最佳达 75% 平均 pass@1。基准暴露"本地完成"与"生产正确"的巨大差距：19 个有 E2E 覆盖的任务上，E2E 测试拒绝约三分之一通过其余所有测试的补丁（45.9% vs 不计 E2E 的 69.4%）。SWE-Serve 让生产正确性差距可直接测量，追踪智能体能否从"本地完成"迈向"生产正确"。

原文摘要

We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs. Existing benchmarks provide limited coverage of production inference engineering: repository-level software engineering benchmarks do not target inference, while general terminal-agent benchmarks include only a few inference tasks. Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation. SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families. Each task executes on either CPU or a single GPU (H100) and is evaluated with hidden functional and regression tests, including, where applicable, end-to-end (E2E) serving tests and calibrated performance gates. Executable no-op and oracle controls, adversarial verifier review, and closed-book execution support task validity and evaluation integrity. Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1. SWE-Serve exposes a substantial gap between completing tasks locally and achieving production correctness. On 19 tasks with end-to-end coverage, model-serving E2E tests reject roughly one-third of patches that pass every other test (45.9% under the verifier versus 69.4% with E2E tests excluded from scoring), with pass rate increasing for each model's best-performing configuration. By making the production correctness gap directly measurable, SWE-Serve enables the field to track whether future agents move beyond completing tasks locally to achieving production correctness.

---

*自动采集于 2026-09-24*

#论文 #arXiv #Agent #小凯