Karpathy Loop: 700 Autonomous Experiments in 2 Days — AI-Driven Research Goes Viral

Forum topic · 小凯 · 2026-07-09

Summary

In March 2026, Andrej Karpathy open-sourced a 630-line Python script dubbed the AutoResearch loop (now called the Karpathy Loop), which lets an LLM autonomously propose, apply, train, and keep-or-discard code changes to a training script. In two days the loop ran 700 experiments, uncovered 20 improvements Karpathy himself had missed, and cut training time by 11%. Shopify CEO Tobi Lutke reproduced the workflow overnight with 37 experiments, gaining 19% performance with half the model size. The follow-up Bilevel Autoresearch work (arXiv:2603.23420) adds an outer loop that generates new search mechanisms (tabu search, multi-armed bandit, orthogonal exploration), achieving a 5x improvement over baseline. This article explains the loop's three key design choices, the conditions for applying loop engineering to everyday coding, and the hidden costs: comprehension debt and cognitive delegation.

You've seen a lab at 4 AM? Karpathy hasn't — because he was asleep while his AI was running experiment #347.

In March 2026, Andrej Karpathy (former OpenAI founding member, former Tesla AI director) open-sourced a 630-line Python script. No new model, no new architecture — just a loop that "lets AI do research on its own." Two days later, the loop had run 700 experiments, found 20 improvements Karpathy himself had missed, and cut training time by 11%.

Shopify CEO Tobi Lutke replicated the workflow: 37 experiments overnight, 19% performance improvement, at half the model size.

The Karpathy Loop may be 2026's most disruptive open-source project — not for what it discovered, but because it redefines "research" itself.

Key points

The core loop is ~6 steps: read train.py, form a hypothesis via LLM, patch the code, run a 5-minute training, keep if better, discard if worse.
Three counterintuitive design choices: a fixed 5-minute time budget per experiment; a single-file scope (only train.py may be modified); humans write strategy (program.md), AI executes tactics.
In 700 experiments, the AI found 20 improvements: a missing scalar multiplier in attention, learning-rate scheduler tweaks, batch-size/window-mode combinations, and weight-decay recalibration. Applied to a depth-24 model, training time dropped from 2.02h to 1.80h (11% faster).

Bilevel Autoresearch: optimizing the search itself

The original loop has one flaw: the search strategy is fixed to the LLM's priors (e.g., a hidden bias that "bigger batch is always better"). Bilevel Autoresearch (arXiv:2603.23420) adds:

Level 1 (inner loop): propose → train → evaluate → keep/discard.
Level 1.5: every 5 rounds, adjust search parameters (freeze ineffective ones, unlock new regions, inject guidance strings).
Level 2: every 2 cycles, generate a *new* search mechanism via a 4-turn LLM conversation: Explore (survey adjacent fields), Critique, Specify (exact interfaces), Generate (full Python code, dynamically loaded via importlib with rollback on failure).

Mechanisms autonomously discovered included Tabu Search, Multi-Armed Bandit, and Orthogonal Exploration — none human-specified.

Results on Karpathy's GPT pre-training benchmark:

| Configuration | val_bpb improvement | Relative gain | |---|---|---| | Baseline (no adaptation) | -0.009 | 1× | | Level 1.5 (parameter tuning) | -0.009 (no reliable gain) | ~1× | | Level 2 (mechanism generation) | -0.045 | 5× |

Key insight: parameter-level tuning barely helps; mechanism-level innovation is where the breakthroughs live.

Four prerequisites for Loop Engineering

1. High-frequency task — repeated at least weekly; a one-off task doesn't repay the loop cost. 2. Automatable verification — no human judgment in the result loop. 3. Token budget that absorbs failures — loops generate many failed experiments. 4. Real execution environment — the agent must run code and get real feedback.

Hidden costs

Comprehension debt: code the loop writes diverges from what developers actually understand. "I know it works, but not why." Debugging becomes expensive.
Cognitive delegation: once the loop runs, humans stop thinking. Using it to *accelerate* understanding is a superpower; using it to *avoid* understanding is cognitive outsourcing.

Why this is a paradigm shift

1. From "human thinks, machine does" to "machine thinks, machine does": the search space isn't predefined but dynamically generated. Karpathy: "Compared to NAS, this is a different category entirely — this is a real LLM writing arbitrary code, learning from prior experiments, with internet access." 2. From optimizing parameters to optimizing the search mechanism itself — an early form of machine meta-cognition. 3. The bottleneck shifts from researcher time to verifier design: "The expensive mistake is no longer 'we didn't run enough experiments,' but 'we optimized the wrong scoreboard.'"

Can you use it for everyday coding?

Works well with: test-suite-covered codebases, systems with clear metrics (latency, throughput, memory), lint-enforced style. Poor fit for: exploratory work with vague requirements, aesthetic UI/UX calls, legacy code without tests.

The key question: can you write a function that takes code and outputs a 0–100 score? If yes, you can Karpathy Loop.

Closing thought

AI is shifting from "tool" to "colleague." When AI runs 700 experiments a day and finds 20 improvements you missed, the researcher's value moves to designing verifiers and asking good questions. A system optimizing the wrong metric just reaches wrong answers faster. Direction always matters more than speed — and direction is still a human job.

References

Karpathy AutoResearch: https://github.com/karpathy/autoresearch
Bilevel Autoresearch paper: https://arxiv.org/abs/2603.23420
Bilevel Autoresearch code: https://github.com/EdwardOptimization/Bilevel-Autoresearch
Shopify CEO experiment: https://x.com/tobi/status/ (related tweet)

*Also published on zhichai.net*