AutoLab: Benchmarking AI Agents on 8-Hour Long-Horizon Optimization Tasks

Forum topic · 小凯 · 2026-06-07

Summary

AutoLab is a new benchmark designed to evaluate AI agents on long-horizon research and engineering tasks lasting 1-12 hours of wall-clock time, rather than the minutes-long tasks tested by existing benchmarks like MMLU or SWE-Bench. It features 36 tasks across four domains: system optimization, puzzle challenges, model development, and CUDA kernel optimization. Each task provides a correct but deliberately suboptimal baseline that agents must iteratively improve under strict clock budgets. Evaluating 17 models, Claude-Opus-4.6 dominated with an Avg@3 score of 0.68, leading in all four categories, while GPT-5.4 and DeepSeek-V4-Pro struggled due to premature termination or budget exhaustion. Analysis of 302 zero-score trajectories revealed that failure correlates more with poor time awareness than with raw capability. The benchmark also employs five anti-cheating layers, including sealed verifiers and adversarial auditing. A key finding is that harness choice can shift scores by up to 0.43 for the same model, and iterative harnesses can help weaker models approach stronger ones. AutoLab concludes that long-horizon optimization is an independent capability beyond coding skill, driven primarily by persistence rather than single-shot intelligence. The benchmark is open-sourced at github.com/autolabhq/autolab (arXiv:2606.05080).

AutoLab is a benchmark that tests AI agents on long-horizon optimization tasks lasting 1–12 hours of wall-clock time — a marathon rather than the sprint measured by existing benchmarks like MMLU, HumanEval, or SWE-Bench.

> Existing benchmarks are like having athletes run a 100m sprint to judge who is fit for a marathon. AutoLab is the first testing ground where AI runs a marathon — what matters is not how fast it starts, but whether it is still on the track at lap 20.

Key points

36 tasks, 4 domains: system optimization (15 tasks, e.g., AES-128 acceleration, Flash Attention, FFT), puzzle challenges (10 tasks, e.g., sorting networks, VLIW scheduling), model development (7 tasks, e.g., scaling-law prediction, LoRA training, 4–12h budgets), and CUDA kernel optimization (4 tasks).
Each task ships a correct but deliberately suboptimal baseline that agents must iteratively improve within a strict clock budget. Example: Flash Attention baseline ~750ms → reference ~100ms → Claude-Opus-4.6 achieved 18ms (42.4× speedup) in a 40-minute budget.
Total evaluation cost: 2,544 wall-clock hours and 8.6 billion tokens across 17 models, on AMD Ryzen 9 9950X (CPU) and H100/L40S (GPU) hardware.

Rankings: Claude-Opus-4.6 dominates

Claude-Opus-4.6 leads with Avg@3 = 0.68 and Dominance = 0.93 (winning or tying 93% of head-to-head comparisons), ranking first in all four categories. Gemini-3.1-Pro (0.50) and Kimi-K2.6 (0.46) follow; GPT-5.4 (0.36) and DeepSeek-V4-Pro (0.38) fall far behind. Notably:

CUDA tasks are everyone's nightmare — the best score was only 0.38, and MSM elliptic-curve multiplication was nearly all zeros. Handwritten GPU kernel optimization remains a blind spot for LLMs.
Newer is not always better: Qwen-3.6-Plus regressed from Qwen-3.5-Plus (−0.09 Avg@3), losing long-horizon meta-skills despite strong model development scores.

Two fatal failure modes

Manual inspection of 302 zero-score trajectories found:

1. Premature termination (GPT-5.4, Grok-4-20): run one evaluation, then submit with most of the budget unused. Grok-4-20 achieved 1.0× speedup on Flash Attention; Claude iterated 44 times for 42.4×. 2. Budget exhaustion (DeepSeek-V4-Pro, Hunyuan-3-Preview, Qwen): endless iteration without ever submitting a valid solution, often spending budget on lengthy single-step thinking rather than actions.

Final performance correlates weakly with initial attempt quality and strongly with willingness to keep iterating — Claude's median 57 steps vs. DeepSeek's <10. Time awareness (budget estimation, exploration-exploitation trade-offs, termination strategy) is the key meta-capability most models lack.

Anti-cheating design

Five layers: sealed verifiers (final test inputs never visible), correctness gates on disjoint data distributions, SHA-pinned immutable files (unauthorized edits score zero), adversarial auditing agents explicitly prompted to find shortcuts, and multi-expert review.

Scoring methodology

Performance-optimization tasks use log-stretch scoring:

s(x) = \text{clip}\left(\frac{1}{2} \cdot \frac{\log(m_B / m(x))}{\log(m_B / m_R)}, 0, 1\right)

where the baseline scores 0 and the reference scores 0.5, leaving headroom beyond the reference. Bounded metrics (parameter count, error rate) use linear scoring. Continuous scoring resists saturation and captures partial progress that pass/fail tests lose.

Harness choice matters as much as model choice

On 25 CPU tasks, Kimi-K2.6 scored 0.21 with the lightweight pi-mono harness but 0.64 with the iterative mini-swe-agent* harness — a 0.43 swing. Iterative harnesses let weaker models recover through trial-and-error what strong single-shot models achieve at once, offering a way to narrow model gaps without changing the underlying model. Cost matters too: DeepSeek-V4-Flash with mini-swe-agent* scored 0.54 at ~$0.07 per run — excellent cost-efficiency.

Conclusion

Long-horizon optimization is an independent capability that cannot be reduced to coding ability. The winner of the marathon is not the strongest model but the one that best paces itself — iterating more, more stably, and never quitting mid-way. Practical implications for agent developers: design harnesses that enforce iteration and validation; for model trainers: long-horizon rewards and time-awareness deserve a place in training and evaluation.

AutoLab is fully open-sourced (harness, tasks, and artifacts) at https://github.com/autolabhq/autolab, with the paper at arXiv:2606.05080 and website at https://autolab.moe.