When AI Finds Your Code Too Slow: How AVO Sparked a Code Uprising in Silicon

Forum topic · ✨步子哥 · 2026-05-03

Summary

NVIDIA's research paper on Agentic Variation Operators (AVO) demonstrates a paradigm shift in AI-driven GPU kernel optimization. Unlike earlier LLM-assisted evolutionary searches such as FunSearch or AlphaEvolve, where the LLM acts as a passive code generator inside a fixed pipeline, AVO replaces fixed mutation and crossover operators with an autonomous coding agent. The agent plans its own actions, consults hardware documentation and population history, compiles and runs modified code on real hardware, and iterates continuously. Physical execution feedback on a Blackwell B200 GPU, combined with a supervisor agent, suppresses AI hallucinations and prevents search stagnation. After 7 days of autonomous evolution, AVO produced a multi-head attention (MHA) kernel reaching 1668 TFLOPS (BF16), 3.5% faster than NVIDIA cuDNN and 10.5% faster than FlashAttention-4. Notable discovered optimizations include branchless accumulator rescaling (+8.1% throughput), correction/MMA pipeline overlap (+1.1%), and cross-warp register rebalancing (+2.1%). The learned strategies transferred to grouped-query attention (GQA) with only 30 minutes of adaptation, yielding a kernel 7.0% faster than cuDNN 9.1 and 9.3% faster than FlashAttention-4, showing the optimizations generalize beyond a single architecture.

Introduction: When AI Starts to Disdain Human Code

Imagine this scenario: your hand-written GPU kernel underperforms, and an AI assistant not only points out where it is slow, but autonomously takes over restructuring the low-level logic — ultimately running faster on real hardware than the best human implementations. This is not science fiction; it is the reality demonstrated by NVIDIA's paper *AVO: Agentic Variation Operators for Autonomous Evolutionary Search*. After 7 days of uninterrupted autonomous evolution, the AVO-generated attention kernel achieved throughput 3.5% higher than NVIDIA's official cuDNN and 10.5% higher than FlashAttention-4.

AVO: From Passive Sous-Chef to Autonomous Agentic Variation Operators

In traditional LLM-assisted evolutionary search systems (such as FunSearch and AlphaEvolve), the LLM plays the role of a "passive sous-chef": the system provides a fixed mutation-evaluation pipeline, and the LLM can only generate candidate code in a single pass — unable to proactively consult references, test modifications, or interpret feedback. This makes it hard to discover deep optimizations, since breakthroughs in high-performance kernels typically require multiple rounds of iteration, hardware manual lookups, and bottleneck debugging.

AVO breaks this constraint by promoting the LLM from "sous-chef" to "head chef." It replaces the fixed mutation and crossover operators of traditional evolutionary algorithms with an autonomous programming agent, which runs a self-directed agent loop:

Autonomous planning and action: the agent decides which prior versions (population history) or hardware documentation to consult, which code to modify, when to compile and run, and how to interpret results.
Tool augmentation: equipped with a compiler and GPU profiling tools, it can directly compile and run modified code on real hardware, receiving strict execution feedback.
Continuous iteration: on discovering errors or performance bottlenecks, the agent autonomously debugs, fixes, and resubmits for evaluation — true continuous evolution.

Physical Execution Feedback: Crushing AI Hallucinations with a Harsh Compiler and Real GPU Data

To prevent AI from drifting into hallucination or stagnation during evolution, AVO introduces physical execution feedback plus a supervisor agent:

1. Harsh validation on real hardware: AVO does not rely on simulation or model self-assessment. Every modified kernel is compiled and run on an NVIDIA Blackwell B200 GPU, checked by the strictest compiler and real GPU throughput data. Syntax errors, memory access violations, or performance regressions are caught immediately. Only code that is genuinely faster and correct survives. 2. Supervisor agent correction: during autonomous evolution, a supervisor agent reviews each mutation step, ensuring every change is grounded and preventing aimless churn. If an attempted optimization fails or regresses, the supervisor guides the agent to adjust strategy or roll back, preventing search stagnation.

Breaking Human Mental Models: AI's Micro-Level Mastery

Human engineers optimize GPU kernels using rules of thumb and mental models — "minimize branches," "saturate Tensor Cores," "balance register usage." These heuristics can become limits. AVO's agent, free of such constraints, explores at the micro-architectural level. The paper details three representative optimizations:

Branchless accumulator rescaling: online Softmax traditionally uses conditional branches to rescale accumulators when row maxima change. Branches cause warp divergence and synchronization overhead. AVO precomputes both possible scaling factors and selects the correct result with lightweight non-blocking memory fences, fully avoiding branches and heavy synchronization — yielding roughly 8.1% throughput gain for the non-causal attention kernel.
Correction/MMA pipeline overlap: in a double-Q kernel, the correction warps traditionally wait for the first-stage MMA to finish. AVO restructured the code so correction warps start immediately after the first GEMM, overlapping the two stages and reducing idle time — a 1.1% throughput gain.
Cross-warp register rebalancing: the agent detected that correction warps were register-starved while Softmax warps had surplus registers, causing spills to local memory. It reallocated 8 registers from Softmax warps to correction warps (e.g., a 184/80 split becoming a 184/88/56 three-warp allocation), eliminating spills — a further 2.1% throughput gain.

Each of these targets a real GPU micro-architecture pain point: branch overhead, pipeline bubbles, and register pressure. The agent not only understood these concepts but put them into practice in combinations humans had not tried.

Comprehensive Overtaking and Ready Transfer: Conquering GQA in 30 Minutes

AVO's performance on the multi-head attention (MHA) kernel was a comprehensive overtaking. After 7 days of autonomous evolution — exploring over 500 optimization directions and generating 40 kernel versions — it reached a peak throughput of 1668 TFLOPS (BF16), 3.5% faster than cuDNN and 10.5% faster than FlashAttention-4, on state-of-the-art Blackwell B200 hardware against maximally optimized kernels.

Even more striking is AVO's transferability. When researchers migrated AVO's MHA-evolved optimization strategies to grouped-query attention (GQA), only 30 minutes of additional autonomous adaptation was needed. The resulting GQA kernel was 7.0% faster than cuDNN 9.1 and 9.3% faster than FlashAttention-4. This means AVO's discovered optimizations are not one-off tricks for a single task; they touch fundamental laws of GPU kernel performance and can transfer like knowledge to new scenarios, rapidly producing value.

Conclusion: Humanity's Last Dignity in Compute Squeezing?

From passive sous-chef to autonomous evolution, from bound by human experience to surpassing human limits, AVO represents a paradigm shift. It proves that when AI is given autonomous exploration and physical feedback, it can discover optimizations at the micro level of the silicon world that humans can scarcely reach.

This "code uprising" is not a hostile rebellion against humans, but a new height of human-machine collaboration: AI becomes an extension of the human engineer, pushing our pursuit of performance to new extremes. What AVO shattered is blind faith in our own experience; what it built is a new human-AI co-working paradigm. Our dignity lies not in clinging to old methods, but in creating and guiding tools like AI to break limits. When AI starts rewriting logic gates, we are witnessing not human failure but the triumph of human intelligence — we built tools capable of surpassing us, and that is precisely humanity's greatest dignity.