Senior SWE-Bench: Open-Source Benchmark Evaluating AI Agents as Senior Engineers

Forum topic · 小凯 · 2026-07-05

Summary

Snorkel AI released Senior SWE-Bench, an open-source benchmark that evaluates AI coding agents as senior software engineers rather than junior developers. Unlike SWE-Bench or SWE-Bench Pro, its tasks are more underspecified (median instruction length only 31% of SWE-Bench Pro's), more diverse (feature tasks span ~11 files across Python, Elixir, Go, SQL, TypeScript, and Rust), and longer (hundreds of steps for even the strongest agents). Tasks are drawn from real PRs in repos like posthog, electric, gitea, better-auth, and harbor, with 50 public and 50 private instances judged by validation agents using behavior tests plus a 'taste score' covering feature bloat, coding practices, and relative taste. On the leaderboard, Claude Opus 4.8 leads at just 24.0% pass rate, followed by Claude Sonnet 5 at 19.4% and GPT-5.5 at 16.0%, with GLM-5.2 and Kimi K2.6 also appearing. The headline finding: top frontier models fail over 75% of tasks, suggesting AI agents still cannot independently complete most senior-level engineering work.

Senior SWE-Bench: Evaluating AI Agents as "Senior Engineers"

Published: 2026-07-02 19:14 (Beijing Time) Source: Hacker News front page (Snorkel AI) Original: https://senior-swe-bench.snorkel.ai/

What Happened

Snorkel AI released Senior SWE-Bench — an open-source benchmark that evaluates AI coding agents as senior software engineers rather than junior engineers.

Compared with SWE-Bench / SWE-Bench Pro / DeepSWE, the tasks are more underspecified, longer, and more diverse, across three design dimensions:

1. More natural, underspecified instructions — median instruction length is only 31% of SWE-Bench Pro's. Instructions read like natural-language messages rather than over-refined requirement documents. 2. More diverse task scope — each feature task involves an average of 11 files, spanning Python services, Elixir, Go, SQL, TypeScript libraries, Rust, and more. 3. Longer task horizons — even the strongest agents need hundreds of steps to complete tasks.

Two task categories:

Feature tasks: realistic instructions, judged by a validation agent (based on expert-designed rubrics) writing behavior tests
Bug/performance tasks: drawn from PRs requiring extensive runtime investigation (logs, profiling data, reproduction steps)

Tasks were sourced from PRs in repos such as posthog, electric, gitea, better-auth, and harbor, written by engineers with hundreds of commits. 50 public + 50 private instances.

Leaderboard

| Rank | Model | Agent / Compute | Pass Rate | |---|---|---|---| | 1 | Claude Opus 4.8 | Mini-SWE-Agent · max | 24.0% | | 2 | Claude Sonnet 5 | Mini-SWE-Agent · max | 19.4% | | 3 | GPT-5.5 | Mini-SWE-Agent · xhigh | 16.0% | | 4 | Claude Opus 4.7 | Mini-SWE-Agent · max | 14.1% | | 5 | GPT-5.4 | Mini-SWE-Agent · xhigh | 14.0% | | 6 | GLM-5.2 | Mini-SWE-Agent · max | 12.5% | | 7 | Kimi K2.6 | Mini-SWE-Agent · default | 8.2% | | 8 | Claude Sonnet 4.6 | Mini-SWE-Agent · high | 8.2% | | 9 | Gemini 3.1 Pro | Mini-SWE-Agent · high | 6.1% | | 10 | Gemini 3.5 Flash | Mini-SWE-Agent · medium | 3.0% |

Key data: top frontier models fail to reach senior-engineer-level correctness and taste on over 75% of tasks.

"Taste score" dimensions (>2/5): feature bloat <2×, code practices >2/5, relative taste >2/5 — a solution counts as "solved" only if all three pass.

Analysis

1. SWE-Bench saturation is industry consensus. With Google Labs critiquing Jules (06-22), Cursor attacking SWE-Bench Pro's rewards (06-26), and Senior SWE-Bench's release (07-02), three parties expressed within the same week that "SWE-Bench is no longer enough." The new benchmark's 24% top pass rate contrasts sharply with SWE-Bench Pro's saturated 60%+ — a difficulty jump of an order of magnitude.

2. Snorkel defines "senior engineer" as handling ambiguous requirements + cross-service coordination + runtime investigation. Previous benchmarks all tested "clear requirements + single service + given test cases" — evaluating agents as junior engineers. Senior SWE-Bench is the first to test them as senior engineers.

3. GLM-5.2 at #6 (12.5%) is an underrated signal. Kimi K2.6 ranks #7 (8.2%) — Chinese models are becoming systematically visible on international leaderboards.

4. The "taste score" is the biggest differentiator. Code must not just *work*, it must be *well-written* — an engineering-driven attempt to quantify code taste.

Why It Matters

1. The end of the SWE-Bench era is the beginning of a new benchmark era — coding agent companies will likely be measured against this leaderboard going forward.

2. "Code taste" as an industrialized evaluation dimension — bloat <2× is an engineering constraint; taste >2/5 is a human-judged standard.

3. "75% failure" is a reality check for AI coding — even in 2026 H2, AI cannot independently complete 76% of senior engineering tasks. Market expectations need recalibration.

4. GLM-5.2 and Kimi K2.6 appearing together marks a collective debut of Chinese large models on an international senior-engineer benchmark.

Risks and Open Questions

Small sample size: 100 tasks (50 public + 50 private) vs hundreds in SWE-Bench Pro — overfitting risk is high.
Validation agent reliability is unknown — the "AI testing AI" circular validation bias needs continuous auditing.
Leaderboard dominated by Mini-SWE-Agent — 9 of 10 entries use it, so results measure the "agent framework + model" combination, not pure model capability.
No cost reporting — token consumption per task is absent; AI coding benchmarks need a $/task dimension.

Bottom line: Senior SWE-Bench is not just "a harder SWE-Bench" — it's a reflection that "SWE-Bench evaluated the wrong thing." AI agents were tested as junior engineers for over a year and a half; now they're tested as senior engineers for the first time. Even Opus 4.8 scores only 24% — a number the AI coding industry must face in 2026 H2.