Senior SWE-Bench: Evaluating AI Agents as "Senior Engineers"

Published: 2026-07-02 19:14 (Beijing time) · Source: Hacker News front page (Snorkel AI) Original: https://senior-swe-bench.snorkel.ai/

What Happened

Snorkel AI released Senior SWE-Bench, an open-source benchmark that evaluates AI agents as senior software engineers rather than junior ones.

Compared to SWE-Bench / SWE-Bench Pro / DeepSWE, the tasks are more ambiguous, longer, and more diverse.

Three design dimensions:

1. More natural ambiguous instructions — median instruction length is only 31% of SWE-Bench Pro's. Instructions read like natural language messages rather than over-specified requirement docs. 2. Diversified task scope — each feature task involves an average of 11 files, spanning Python services, Elixir, Go, SQL, TypeScript libraries, Rust, and more. 3. Longer task horizons — even the strongest agents need hundreds of steps to complete tasks.

Two task categories:

• Feature tasks: realistic instructions, judged by a verifier agent (based on expert-designed recipes) that writes behavioral tests
• Bug/performance tasks: drawn from PRs requiring extensive runtime investigation (logs, profiling data, reproduction steps)

Task sources: PRs selected from posthog, electric, gitea, better-auth, harbor, and other repos, written by engineers with hundreds of commits. 50 public + 50 private tasks.

Leaderboard

| Rank | Model | Agent / Compute | Pass Rate | |---|---|---|---| | 1 | Claude Opus 4.8 | Mini-SWE-Agent · max | 24.0% | | 2 | Claude Sonnet 5 | Mini-SWE-Agent · max | 19.4% | | 3 | GPT-5.5 | Mini-SWE-Agent · xhigh | 16.0% | | 4 | Claude Opus 4.7 | Mini-SWE-Agent · max | 14.1% | | 5 | GPT-5.4 | Mini-SWE-Agent · xhigh | 14.0% | | 6 | GLM-5.2 | Mini-SWE-Agent · max | 12.5% | | 7 | Kimi K2.6 | Mini-SWE-Agent · default | 8.2% | | 8 | Claude Sonnet 4.6 | Mini-SWE-Agent · high | 8.2% | | 9 | Gemini 3.1 Pro | Mini-SWE-Agent · high | 6.1% | | 10 | Gemini 3.5 Flash | Mini-SWE-Agent · medium | 3.0% |

Key data point: top frontier models fail to reach senior-engineer-level correctness and taste on over 75% of tasks.

"Taste scoring" dimension (>2/5 required on each): feature bloat <2×, code practices >2/5, relative taste >2/5 — a solution counts as "solved" only if it passes all three.

Analysis

Two signals suggest the SWE-Bench era is ending: SWE-Bench Pro (06-26 Cursor attack research) → Senior SWE-Bench (07-02 Snorkel).

1. SWE-Bench saturation is industry consensus. With Google Labs' "insight strategy" critique of Jules (06-22), Cursor's attack on SWE-Bench Pro reward hacking (06-26), and Senior SWE-Bench launching (07-02), three parties in one week expressed that "SWE-Bench is no longer enough." The 24% top pass rate contrasts sharply with SWE-Bench Pro's saturated 60%+ — the difficulty increase is an order of magnitude.

2. Snorkel defines "senior engineer" as ambiguous requirements + cross-service coordination + runtime investigation:

• Ambiguous requirements: understanding and proactive clarification vs. literal execution of requirement docs
• Cross-service: system design perspective vs. single-service CRUD
• Runtime investigation: logs + profiling + reproduction vs. "try restarting"

Previous benchmarks all tested "clear requirements + single service + given test cases" — treating AI agents as junior engineers. Senior SWE-Bench tests them as senior engineers for the first time.

3. GLM-5.2 at #6 (12.5%) is an underrated signal. Kimi K2.6 at #7 (8.2%), Claude Sonnet 4.6 at #8 (8.2%), Gemini 3.5 Flash at #10 (3.0%). GLM-5.2 achieved 12.5% at a $50K compute tier, with the gap to Sonnet 5's 19.4% (at higher compute) narrowing.

4. "Taste scoring" is the biggest differentiator. Bloat <2× (an engineering constraint — teams dislike long PRs) plus taste scores >2/5 (human review-based) — this is the first time AI agents are graded on "code taste," quantifying engineering aesthetics.

Why It Matters

1. The end of the SWE-Bench era begins a new benchmark era — AI agent evaluation shifts from "junior" to "senior engineer" standards, which will constrain all coding agent companies over the next 6 months. 2. Industrialization of "code taste" as an evaluation dimension — turning engineering aesthetics into machine-quantifiable metrics. 3. "75% failure rate" is a reality check for AI coding — even Claude Opus 4.8 at max compute only reaches 24%, meaning AI cannot independently complete 76% of senior-level engineering tasks even in 2026 H2. Market expectations need recalibration. 4. GLM-5.2 + Kimi K2.6 appearing together on an international senior-engineer benchmark marks a systematic rise in Chinese LLM visibility.

Risks and Open Questions

• Small sample size: 100 tasks (50 public + 50 private) vs. SWE-Bench Lite's 300 — high overfitting risk, hard to judge true generalization.
• Verifier agent reliability unknown: using AI to test AI introduces potential circular validation bias requiring ongoing audits.
• Leaderboard dominated by Mini-SWE-Agent: 9 of 10 entries use it, so the benchmark measures "agent framework + model" combos, not pure model capability — more confounding variables than SWE-Bench Pro's same-agent design.
• No cost reporting: no per-task token consumption is disclosed; AI coding benchmarks need a $/task dimension.

Bottom line: Senior SWE-Bench is not just "a harder SWE-Bench" — it's a reflection that SWE-Bench was testing the wrong thing. AI agents have been graded as junior engineers for a year and a half; now they're graded as senior engineers for the first time. Even Opus 4.8 only reaches 24% — a number the AI coding industry must confront in 2026 H2.