AutoLab: Benchmarking AI Agents on 8-Hour Marathon Optimization, Not 8-Minute Sprints

Forum topic · 小凯 · 2026-06-07

Summary

AutoLab is a new benchmark designed to test whether frontier AI models can perform long-horizon auto research and engineering tasks lasting 1-12 hours, rather than single-shot coding or Q&A. Spanning 36 tasks across four tracks—systems optimization, algorithmic puzzles, ML model development, and CUDA kernel optimization—the benchmark evaluates 17 models under strict wall-clock budgets with continuous, calibrated scoring and five layers of anti-cheating protections (sealed verifiers, correctness gates, SHA-pinned files, adversarial auditing, and expert review). Results show Claude-Opus-4.6 dominating with an Avg@3 score of 0.68, leading all four categories, while GPT-5.4 and DeepSeek-V4-Pro underperform due to premature termination or budget exhaustion. Analysis of 302 zero-score trajectories reveals that persistence and time-budget awareness—not initial reasoning quality—are the strongest predictors of success. The benchmark also found harness choice can swing scores by up to 0.43, and that handwritten GPU kernel optimization remains a blind spot for all models. AutoLab is open-sourced with all tasks, harnesses, and artifacts.

AutoLab: When AI Must Work for 8 Hours Instead of 8 Minutes, Who Is the Real Winner?

> Existing benchmarks are like asking athletes to sprint 100 meters and then judging who is fit for a marathon. AutoLab is the first test track that makes AI run a marathon — measuring not how fast it starts, but whether it is still on the track at lap 20. The results are surprising: sprint champions like GPT-5.4 and DeepSeek-V4-Pro either sat down halfway or collapsed before the finish. The true winner was not the strongest, but the one that best understood pacing.

Key points

AutoLab is a benchmark for ultra-long-horizon AI agent evaluation: 36 tasks, 4 domains, 1–12 hour strict wall-clock budgets, 17 models evaluated over 2,544 compute hours and 8.6 billion tokens.
Claude-Opus-4.6 dominates with Avg@3 = 0.68 and Dominance = 0.93, ranking first in all four categories. Gemini-3.1-Pro is second (0.50); GPT-5.4 (0.36) and DeepSeek-V4-Pro (0.38) trail badly.
Long-horizon optimization is an independent capability not reducible to one-shot coding skill — persistence matters more than per-step intelligence.
Harness choice can swing scores by 0.43: Kimi-K2.6 scored 0.21 with pi-mono but 0.64 with an iterative mini-swe-agent harness.
CUDA kernel optimization is a universal blind spot — the best score in that category was only 0.38.

Why AutoLab: the blind spot of existing benchmarks

| Test type | Examples | Duration | Problem | |:---|:---|:---|:---| | Static QA | MMLU, GSM8K | seconds | knowledge, not action | | Single-shot coding | HumanEval, MBPP | minutes | one-shot correctness only | | Short-horizon agents | SWE-Bench, OSWorld | minutes–hours | one task, no sustained iteration | | Long-horizon | AlphaEvolve, AutoResearch | hours | tightly coupled to specific models/tools |

AutoLab's three design principles: ultra long-horizon (strict clock budgets), continuous and calibrated scoring across heterogeneous metrics (runtime, perplexity, parameter count), and hack-resistance.

The 36 tasks: four tracks

| Category | Tasks | Challenge | Budget | |:---|:---:|:---|:---:| | Systems optimization | 15 | performance maximization (AES-128, Flash Attention, FFT, hash join) | 1–4h | | Puzzle challenges | 10 | algorithmic/combinatorial optimization (sorting networks, VLIW scheduling) | 1–8h | | Model development | 7 | ML training/tuning (scaling-law prediction, OCR fine-tuning, LoRA, GRPO) | 4–12h | | CUDA kernels | 4 | handwritten GPU kernel optimization (Huffman decode, ICP, NTT, MSM) | 1–4h |

Each task ships a correct but deliberately suboptimal baseline. Example — Flash Attention: baseline ~750ms, reference ~100ms; Claude-Opus-4.6 reached 18ms (42.4× speedup) within a 40-minute budget.

Rankings and domain results

| Rank | Model | Avg@3 | Dominance | |:---:|:---|:---:|:---:| | 1 | Claude-Opus-4.6 | 0.68 | 0.93 | | 2 | Gemini-3.1-Pro | 0.50 | 0.62 | | 3 | Kimi-K2.6 | 0.46 | 0.62 | | 4 | MiMo-V2.5-Pro | 0.45 | 0.53 | | 5 | GLM-5 | 0.43 | 0.57 | | 6 | DeepSeek-V4-Pro | 0.38 | 0.47 | | 7 | GPT-5.4 | 0.36 | 0.39 | | 8 | Grok-4-20 | 0.35 | 0.42 | | 9 | Hunyuan-3-Preview | 0.31 | 0.34 | | 10 | MiniMax-M2.7 | 0.27 | 0.28 | | 11 | Qwen-3.6-Plus | 0.27 | 0.32 |

Notable findings:

CUDA is everyone's nightmare: MSM Pippenger defeated nearly all models; only Claude scored meaningfully on Huffman Decode (0.45) and ICP (0.55).
Qwen-3.6-Plus regressed vs 3.5-Plus (−0.09 Avg@3): despite strong model development (0.88), CUDA/puzzle/systems scores collapsed — new versions can lose long-horizon meta-capabilities.

Anti-cheating: five layers of defense

1. Sealed verifiers — final test inputs/outputs invisible during development. 2. Correctness gates — metrics only count if held-out correctness thresholds pass (e.g., generic VQA accuracy drop >10% → score zeroed). 3. SHA-pinned immutable files — unauthorized modification → immediate zero. 4. Adversarial auditing — dedicated adversarial agents explicitly prompted to exploit shortcuts; exploitable tasks are patched or removed. 5. Multi-expert review — ≥2 independent experts plus a format-audit agent per task.

Scoring methodology

For unbounded performance metrics, a log-stretched score linear in log space, with baseline = 0, reference = 0.5:

s(x) = \text{clip}\left(\frac{1}{2} \cdot \frac{\log(m_B / m(x))}{\log(m_B / m_R)}, 0, 1\right)

For bounded metrics, linear scoring (baseline = 0, reference = 1):

s(x) = \text{clip}\left(\frac{m_B - m(x)}{m_B - m_R}, 0, 1\right)

Continuous scoring resists saturation and preserves diagnostic information that pass/fail would discard.

Core finding: lack of time awareness is fatal

Analysis of 302 zero-score trajectories revealed two dominant failure modes:

Mode A — premature termination: run the evaluator once, then submit, leaving budget unused. Victims: GPT-5.4, Grok-4-20 (Grok ran one evaluation, 1.0× speedup; Claude iterated 44 times to 42.4×).
Mode B — budget exhaustion: iterating until timeout without ever submitting a valid solution. Victims: DeepSeek-V4-Pro, Hunyuan-3-Preview, Qwen-3.6-Plus (DeepSeek spent budget on long single-step thinking rather than command execution).

Failure breakdown of the 302 zero trajectories: ~40% timeout/context exhaustion, ~35% capability gap, ~15% instruction violations, ~10% other (server/format/sandbox issues).

Median steps per run: Claude-Opus-4.6 = 57 (score 0.68); Gemini-3.1-Pro = 12 (0.50); DeepSeek-V4-Pro <10 (0.38). Final performance correlates weakly with initial attempt quality and strongly with willingness to keep iterating. Three required meta-skills: budget estimation, exploration–exploitation trade-off, and early-termination judgment.

Harness effects: same model, different fate

On 25 CPU tasks, three harnesses were compared:

| Harness | Kimi-K2.6 | GPT-5.4 | |:---|:---:|:---:| | terminus-2 (default) | 0.46 | 0.36 | | pi-mono (lightweight) | 0.21 | 0.50 | | mini-swe-agent* (iterative) | 0.64 | 0.33 |

The iterative harness's system prompt mandates: never submit on the first pass, iterate after every change (build → run local verifier → check metric → continue or roll back), and only submit after exhausting reasonable optimization ideas. Iterative harnesses let weaker models recover through trial-and-error what stronger models solve in one shot; lightweight harnesses maximize strong one-shot models. Harness design itself is a lever for closing model gaps. Cost matters too: DeepSeek-V4-Flash + mini-swe-agent* scored 0.54 at ~$0.07 per run — excellent cost-efficiency.

Limitations and outlook

Scope: executable systems engineering and ML workflows, not general scientific discovery.
Execution-dependent: multi-hour runs couple results to hardware/environment (AMD Ryzen 9 9950X for CPU tasks; H100/L40S via Modal for GPU).
Cost: 2,544 wall-clock hours and 8.6B tokens — hard for a single team to replicate.

Future directions: training time-aware models, adaptive harnesses, specialized tooling for CUDA blind spots, and cheaper evaluation.

Conclusion

Long-horizon optimization is an independent capability that cannot be reduced to coding ability. Claude-Opus-4.6 won not because it is smarter, but because it knows how to run a marathon — when to push, when to persist, and when not to quit. For agent builders: design harnesses that encourage iteration; for model trainers: long-horizon RL and time-awareness deserve a place in training and evaluation.

References

Paper: *AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?* (arXiv:2606.05080)
Code: https://github.com/autolabhq/autolab
Website: https://autolab.moe
Default harness: terminus-2 (Harbor framework)