Transformer-Guided Swarm Intelligence for Frugal Neural Architecture Search on Consumer GPUs

Forum topic · 小凯 · 2026-07-15

Summary

This paper proposes a frugal, memetic Neural Architecture Search (NAS) framework that democratizes deep learning model design on consumer-grade hardware, where traditional NAS typically demands thousands of GPU-days. The method couples an autoregressive Transformer controller trained via reinforcement learning for global macro-search with the local micro-exploitation of an Artificial Bee Colony (ABC) algorithm. A dynamic entropy mechanism forces topological exploration when performance stagnation is detected, preventing premature convergence during the RL phase. Evaluated on a single NVIDIA RTX 3060, the hybrid approach resolves the cold-start problem inherent to metaheuristics. By penalizing network depth, the framework actively mitigates model bloat: on CIFAR-10 it discovers an efficient architecture achieving 84.85% accuracy with roughly 174,000 parameters—substantially smaller than baselines like ResNet-20—within a 3-hour search. The framework also demonstrates flexibility on credit card fraud detection, directly optimizing F1-Score on highly imbalanced tabular data and reaching 0.71 with a compact network of about 4,600 parameters. Paper: arXiv 2607.11826, author Romain Amigon.

Paper Overview

Field: Machine Learning
Author: Romain Amigon
Published: 2026-07-13
arXiv: 2607.11826

Abstract (translated)

Neural Architecture Search (NAS) has automated the design of deep learning models but traditionally requires massive computational resources, often measured in thousands of GPU-days. This paper proposes a frugal and memetic NAS framework designed to democratize architecture design on consumer-grade hardware.

The approach combines the global macro-search capabilities of an autoregressive Transformer controller, trained via Reinforcement Learning (RL), with the local micro-exploitation of an Artificial Bee Colony (ABC) algorithm. To prevent premature convergence during the RL phase, a dynamic entropy mechanism forces topological exploration upon detection of performance stagnation.

Key Findings

Hardware: Evaluated on a single consumer GPU (NVIDIA RTX 3060), effectively resolving the 'cold-start' problem inherent to metaheuristics.
Anti-bloat: By penalizing network depth, the framework actively mitigates model bloat.
CIFAR-10: Discovered an efficient architecture with ~174,000 parameters achieving 84.85% accuracy—significantly smaller than standard baselines such as ResNet-20—with a search time of only 3 hours.
Tabular data: Applied to credit card fraud detection, directly optimizing F1-Score on highly imbalanced data, achieving 0.71 with a compact network of ~4,600 parameters.

The results demonstrate that hybrid RL + swarm intelligence NAS can deliver competitive, compact architectures at a fraction of the usual computational cost.

---

*Auto-collected on 2026-07-15*