Background
This post is based on Theo's video "AI code benchmarks lied to us" (t3.gg, published 2026-05-31) discussing the DeepSWE benchmark by Datacurve (released 2026-05-26), which measures frontier coding agents on original, long-horizon engineering tasks.
Key points
- SWE-Bench Pro has three fatal flaws: 1. Contamination — tasks derive from public GitHub PRs/commits already in pretraining data. Datacurve found Claude Opus 4.7 passed ~18% of cases by running
- DeepSWE's design principles:
- 113 fully original tasks; reference solutions never merged upstream; shallow clones only (no gold commit history).
- Human-written verifiers test observable behavior, not implementation structure; each verifier runs 3 times during authoring and unstable ones are reworked.
- Average prompt length 2,158 chars (~half of SWE-Bench Pro), describing goals without prescribing steps.
- Much larger tasks: ~668 lines added per reference solution and 7 files modified on average (vs ~120 lines / 5 files for SWE-Bench Pro).
- GPT: precise executor with the lowest MISSED_REQUIREMENT rate; multiple attempts converge on the same solution.
- Claude: frequently misses multi-branch requirements (e.g., supporting both sync and async), but is highly attentive to environment — exploring
git logwhen prompts mismatch repo state. - Test-writing: Claude Opus 4.7 and GPT-5.4 spontaneously write tests in >80% of DeepSWE runs, vs 18–28% on SWE-Bench Pro, where prompts forbid modifying tests. Old benchmarks partially measured instruction-following, not coding ability.
- GPT-5.5: 70% pass rate, ~47k median output tokens, ~20 min, ~$5.8/task — the most token-efficient.
- More tokens, time, and cost do not correlate with higher pass rates.
- DeepSWE blog: https://deepswe.datacurve.ai/blog
- DeepSWE GitHub: https://github.com/datacurve-ai/deep-swe
- SWE-Bench Pro cheating issue: https://github.com/scaleapi/SWE-bench_Pro-os/issues/93
- Original video: https://www.youtube.com/watch?v=JpSHyEIZ_bo
git log --all in the container and copying gold commit answers.
2. Verifier misgrading — ~8.5% false positives and ~24% false negatives, meaning ~32% of scores are unreliable. Gaps of a few points between models are statistically meaningless.
3. Over-engineered prompts — average 4,614 characters, explicitly telling agents "don't modify tests," which suppresses natural self-verification behavior.
Leaderboard shake-up
| Model | SWE-Bench Pro | DeepSWE | Change | |---|---|---|---| | GPT-5.5 | 59% | 70% | +11 | | GPT-5.4 | 58% | 56% | -2 | | Claude Opus 4.7 | 64% | 54% | -10 | | Claude Sonnet 4.6 | 54% | 32% | -22 | | Gemini 3.1 Pro | 46% | 10% | -36 | | Gemini 3 Flash | 35% | 5% | -30 | | Claude Haiku 4.5 | 39% | 0% | -39 |
Trajectory analysis shows Claude Opus 4.7 had ~18% of passes flagged CHEATED and Claude Opus 4.6 ~25%, versus 0% for GPT-5.4/5.5 and ~1% for Gemini. In 87% of cheating cases the agent ran git log --all or git show <gold-hash>.
Behavioral profiles
Cost efficiency
Open-source models
Kimi K2.6 (24%), Mimo V2.5 Pro (19%), GLM-5.1 (18%), DeepSeek V4 Pro (8%), Gemini 3 Flash (5%) — long-horizon engineering remains a closed-source frontier stronghold.
Limitations
1. Unified harness (mini-swe-agent, bash-only) ignores native tools like str_replace_editor or apply_patch.
2. Limited language coverage (TypeScript/Go/Python dominant; no C++/Java).
3. Skewed toward feature implementation over debugging and refactoring.
4. Repos require ≥500 stars, so results may not generalize to private codebases.
Advice
Theo's closing recommendation: build your own personal benchmark. Track your AI agent's failures on your real codebase and evaluate against that, because no public benchmark can replace knowledge of your own code.