Overview
This post dissects AIRA (Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design) by Alberto Pepe, Chien-Yu Lin, Despoina Magka, Bilge Acun, Yannan Nellie Wu, Anton Protopopov, Carole-Jean Wu, and Yoram Bachrach (Meta FAIR).
- Paper: https://arxiv.org/abs/2605.15871
- AIRAformers: pure Transformer variants differing from standard Transformers in layer ordering, width ratios, and connectivity patterns
- AIRAhybrids: Transformer + Mamba hybrids where agents found Mamba more efficient than Attention at certain layers
- Configurable setups performed worse: more freedom (tunable hyperparameters) led agents to get lost in oversized search spaces
- Greedy beats one-shot: iterative refinement significantly outperforms one-shot generation
- Opus 4.6 vs Gemini 3 Pro: different base LLMs excel at different tasks, bounding agent capability
- Significant gaps remain on long-range tasks vs human-designed specialists (S4, H3, Mamba)
- Primitives themselves are still human-provided; agents cannot yet invent new computational primitives
- Configurable setups hurt performance; agents can't yet handle high-dimensional continuous spaces
- The 16-layer proxy correlation is imperfect; winners may not stay optimal at 3B
- Non-trivial compute cost (24-hour GPU budgets, multi-agent parallelism)
- No theoretical explanation of why agent designs work; reproducibility variance underreported; humans still build the training infrastructure
- Paper: https://arxiv.org/abs/2605.15871
- Composer (Meta NAS framework): https://arxiv.org/abs/2502.07897
- Long Range Arena: https://arxiv.org/abs/2011.04006
- Mamba (state space model): https://arxiv.org/abs/2312.00752
The core question: since Transformers have dominated deep learning since 2017's *Attention Is All You Need*, would an AI agent designing architectures from scratch choose a Transformer? Traditional NAS searches within human-defined spaces using deterministic strategies (Bayesian optimization, evolution). AIRA asks something more radical: give agents raw computational primitives (Attention, MLP, Mamba, convolution) and let them decide how to combine, order, and scale them.
Key points
Two-layer framework
AIRA-Compose (architecture search), built on an agentic adaptation of Meta's Composer framework: 1. Agentic Exploration — 11 LLM agents work in parallel, freely proposing structural hypotheses that combine primitives into 16-layer small-scale candidates 2. Fast-Proxy Evaluation — quick train/validation on small datasets, scored by validation loss and downstream accuracy 3. Aggregation — layer-wise clustering picks the most frequent primitives, smoothing proxy-training noise 4. Extrapolation — stretching contiguous blocks and stacking the architecture to reach 350M / 1B / 3B parameters
The 16-layer proxy is key: small-scale performance correlates strongly with large-scale results, so agents iterate fast at small scale before winners are scaled up.
AIRA-Design (low-level implementation): 20 agents write code from scratch—novel attention mechanisms for Long Range Arena tasks (ListOps, Text, Retrieval) and training-script optimization (minimizing validation loss under fixed time). Compose searches the macro architecture; Design handles micro implementation.
Outputs: two model families, 14 models
Experimental results (1B parameters, fixed token budget)
| Model | vs Llama 3.2 | vs Nemotron-2 | Note | |:---|:---|:---|:---| | AIRAhybrid-D | +3.8% downstream accuracy | - | Best hybrid | | AIRAhybrid-C | - | +23% scaling performance | Better scaling | | AIRAformer-C | - | - | +54% scaling speed (highest efficiency) |
Agent-designed architectures also show steeper scaling curves—more capability gain per parameter added.
Long Range Arena (Normalized Score, 1.0 = human SOTA):
| Task | Best Agent | Score | Gap | |:---|:---|:---|:---| | ListOps | Greedy Opus 4.6 | 0.51 | -49% | | Text | Greedy Gemini 3 Pro | 0.88 | -12% | | Retrieval | Greedy Opus 4.6 | 0.79 | -21% |
Key findings:
What did agents discover that humans missed?
1. Emergent hybrid architectures — agents spontaneously proposed mixing Attention and Mamba across layers; unlike NAS, the search space itself was agent-invented 2. Heterogeneous layer patterns — instead of uniform stacking, layers at different depths show distinct primitive distributions, suggesting shallow layers do local feature extraction while deep layers do global semantic integration 3. Improved scaling efficiency — possibly a more efficient parameter-to-capability conversion
Limitations (per the paper)
How far are we from "AI researching AI"?
Current AIRA is AI-assisted architecture search: agents propose novel combinations, but primitives, evaluation criteria, infrastructure, and final code review are all human. Next steps require:
1. Primitive discovery — inventing new computational primitives, not just recombining known ones 2. End-to-end loops — design → training code → experiments → analysis → improvement 3. Theoretical explanation — agents deriving why their designs work from first principles 4. Self-improvement — agent-designed architectures training stronger agents (recursive self-improvement)
Context vs related work: traditional NAS (AutoML, DARTS) uses fixed search strategies; Composer (Meta, 2025) is non-agentic; AlphaEvolve (DeepMind, 2025) targets math algorithms; AI Scientist v2 (Sakana, 2025) is more end-to-end but focused on paper generation.
2026 signals across this week's papers (OProver, HarnessX, LFM2.5, AIRA): algorithms beat parameters (32B beating 560B; 350M beating 600M); system-level optimization can outweigh model swaps; and AI-assisted R&D—not AI-replacing R&D—is the realistic path.
AIRA's real value isn't that agents beat humans (they don't, especially on long-range tasks), but that it proves the path is viable: given raw primitives, agents find effective combinations humans never tried. The pivotal question: when will agents invent computational structures beyond Transformers and Mamba? That's when "AI researching AI" truly begins.