Overview
This post dissects AIRA (Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design) by Alberto Pepe, Chien-Yu Lin, Despoina Magka, Bilge Acun, Yannan Nellie Wu, Anton Protopopov, Carole-Jean Wu, and Yoram Bachrach (Meta FAIR).
- Paper: https://arxiv.org/abs/2605.15871
- AIRA-Compose (macro architecture search), built on an agentified version of Meta's Composer framework: 1. Agentic Exploration — 11 LLM agents work in parallel, each proposing structural hypotheses that combine primitives into 16-layer small-scale candidates. 2. Fast-Proxy Evaluation — quick training/validation on small datasets; scoring by validation loss and downstream accuracy. 3. Aggregation — layer-wise clustering selects the most frequently appearing primitives, smoothing proxy-training noise and overfitting. 4. Extrapolation — stretching and stacking winning blocks to target 350M / 1B / 3B parameters.
- The 16-layer proxy is chosen because small-model performance correlates strongly with large-model outcomes.
- AIRA-Design (micro implementation): 20 agents write code from scratch — novel attention mechanisms for Long Range Arena tasks (ListOps, Text, Retrieval) and training-script optimization (Autoresearch: minimize validation loss within a fixed time budget).
- AIRAformers: Transformer variants differing from standard Transformers in layer arrangement, width ratios, and connectivity patterns.
- AIRAhybrids: Transformer + Mamba mixtures — agents discovered that in some layers Mamba (a state-space model) beats Attention, balancing compute efficiency and long-range dependency.
- Emergent hybrid architectures: agents spontaneously proposed mixing Attention and Mamba across layers — a search space they invented, not one humans predefined.
- Heterogeneous layers: rather than uniform stacking, winning models show distinct primitive distributions in early, middle, and late layers, suggesting different depths serve different computational roles.
- Configurable setups perform worse: giving agents more freedom (tunable hyperparameters) degrades results — agents get lost in oversized search spaces.
- Greedy beats one-shot: iterative refinement with reflection is critical to design quality.
- Base model matters: Opus 4.6 does better on ListOps/Retrieval; Gemini 3 Pro on Text — agent capability is bounded by the underlying LLM.
- AIRA paper: https://arxiv.org/abs/2605.15871
- Composer (Meta NAS framework): https://arxiv.org/abs/2502.07897
- Long Range Arena: https://arxiv.org/abs/2011.04006
- Mamba: https://arxiv.org/abs/2312.00752
The central question: since Transformers have dominated since 2017's *Attention Is All You Need*, what would an AI agent design if freed from human-defined search spaces? Traditional NAS constrains AI to human-drawn boundaries (fixed hyperparameter grids, deterministic strategies like Bayesian optimization). AIRA instead gives agents raw computational primitives (Attention, MLP, Mamba, convolution) and lets them decide how to combine, arrange, and scale them.
Key points
Two complementary frameworks
Outputs: two architecture families, 14 models
Results (1B parameters, fixed token budget)
| Model | vs Llama 3.2 | vs Nemotron-2 | Note | |:---|:---|:---|:---| | AIRAhybrid-D | +3.8% downstream accuracy | – | best hybrid | | AIRAhybrid-C | – | +23% scaling performance | better scaling | | AIRAformer-C | – | – | +54% scaling speed, best scaling efficiency |
Agent-designed architectures also show steeper scaling curves — better parameter-to-capability conversion.
Long Range Arena (Normalized Score, 1.0 = human SOTA):
| Task | Best Agent | Score | Gap | |:---|:---|:---|:---| | ListOps | Greedy Opus 4.6 | 0.51 NS | -49% | | Text | Greedy Gemini 3 Pro | 0.88 NS | -12% | | Retrieval | Greedy Opus 4.6 | 0.79 NS | -21% |
Notable findings
Limitations acknowledged
1. Agents remain well below human SOTA on long-range tasks (vs. S4, H3, Mamba). 2. Primitives are still human-provided; agents cannot yet invent new ones. 3. Too much freedom hurts; the 16-layer proxy correlation is imperfect at 3B scale. 4. Non-trivial compute cost (24-hour GPU budgets, parallel agents), no theoretical explanation of *why* designs work, and limited reproducibility reporting.
Where does this leave "AI researching AI"?
AIRA today is AI-assisted architecture search: humans supply primitives, evaluation criteria, training infrastructure, and final code review. Next steps toward true AI-driven research:
1. Primitive discovery — inventing new computational primitives beyond recombining known ones. 2. End-to-end loops — design → training code → experiments → analysis → improvement. 3. Theoretical explanation — deriving design principles from first principles, not just empirical wins. 4. Recursive self-improvement — agent-designed architectures training stronger agents.
Comparison with related work
| Work | Searches | Automation | Relation to AIRA | |:---|:---|:---|:---| | Traditional NAS (AutoML, DARTS) | Hyperparameters/topology | Medium | AIRA replaces fixed strategies with agents | | Composer (Meta, 2025) | Hybrid architectures | High (non-agentic) | AIRA-Compose's agentified basis | | AlphaEvolve (DeepMind, 2025) | Mathematical algorithms | High | Similar idea, different domain | | AI Scientist v2 (Sakana, 2025) | Full research pipeline | Very high | More end-to-end, focused on paper generation |
Trend signals for 2026
Alongside three other papers discussed (OProver, HarnessX, LFM2.5), AIRA points to:
1. Algorithm > parameters: 32B beating 560B, 350M beating 600M, agent designs beating hand-crafted ones. 2. System optimization > model capability: runtime "harness" improvements can outperform model swaps. 3. AI-assisted R&D > AI-replacing R&D: the realistic near-term path is incremental.
AIRA's real value is not that agents beat humans (they clearly don't on long-range tasks), but that the path is proven viable. The key question ahead: can search expand from *recombining known primitives* to *inventing entirely new ones*? That is when "AI researching AI" truly begins.