English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

SWE-Serve: A Benchmark for Agentic Engineering on Production Inference Tasks

Forum topic · 小凯 · 2026-09-24

Summary

SWE-Serve is a benchmark for evaluating AI agents on production inference engineering tasks, such as implementing features across the serving stack of SGLang, including model support, runtime execution, and public APIs. Existing benchmarks poorly cover this domain: repository-level SWE benchmarks do not target inference, terminal-agent benchmarks include few inference tasks, and dedicated inference benchmarks focus on isolated kernel generation rather than repository-scale feature implementation. SWE-Serve provides 53 repository-grounded tasks derived from recent SGLang production changes, spanning six inference engineering families. Tasks run on CPU or a single H100 GPU and are scored with hidden functional and regression tests, end-to-end (E2E) serving tests, and calibrated performance gates; no-op/oracle controls, adversarial verifier review, and closed-book execution ensure validity. Across 11 models and 31 model-effort configurations, the best achieves 75% mean pass@1. Notably, on 19 tasks with E2E coverage, E2E tests reject roughly one-third of patches that pass all other tests (45.9% vs. 69.4% without E2E scoring), exposing a large gap between local task completion and production correctness. Paper: arXiv:2609.26777.

SWE-Serve: Benchmarking Agentic Engineering For Production Inference

Overview

  • Research area: Agents
  • Authors: Jennifer Williams, Dave Farris, Jeff Farris, Jiantao Jiao
  • Published: 2026-09-22
  • arXiv: 2609.26777
  • Abstract

    We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs. Existing benchmarks provide limited coverage of production inference engineering: repository-level software engineering benchmarks do not target inference, while general terminal-agent benchmarks include only a few inference tasks. Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation.

    SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families. Each task executes on either CPU or a single GPU (H100) and is evaluated with hidden functional and regression tests, including, where applicable, end-to-end (E2E) serving tests and calibrated performance gates. Executable no-op and oracle controls, adversarial verifier review, and closed-book execution support task validity and evaluation integrity.

    Key Findings

  • Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1.
  • SWE-Serve exposes a substantial gap between completing tasks locally and achieving production correctness.
  • On 19 tasks with end-to-end coverage, model-serving E2E tests reject roughly one-third of patches that pass every other test (45.9% under the verifier versus 69.4% with E2E tests excluded from scoring), with pass rate increasing for each model's best-performing configuration.
  • By making the production correctness gap directly measurable, SWE-Serve enables the field to track whether future agents move beyond completing tasks locally to achieving production correctness.
---

*Auto-collected on 2026-09-24*

Tags

#benchmark#ai-agents#inference-engineering#sglang#software-engineering#arxiv#llm-evaluation

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178635145