Transformer-Guided Swarm Intelligence for Frugal Neural Architecture Search

Forum topic · 小凯 · 2026-07-15

Summary

This paper proposes a frugal, memetic Neural Architecture Search (NAS) framework that democratizes deep learning model design on consumer-grade hardware, where traditional NAS typically requires thousands of GPU-days. The method combines the global macro-search capability of an autoregressive Transformer controller trained via reinforcement learning with the local micro-exploitation of an Artificial Bee Colony (ABC) algorithm. A dynamic entropy mechanism forces topological exploration when performance stagnation is detected, preventing premature convergence during the RL phase. Evaluated on a single NVIDIA RTX 3060, the hybrid approach resolves the cold-start problem inherent to metaheuristics. By penalizing network depth, the framework actively mitigates model bloat: on CIFAR-10 it discovers an efficient architecture achieving 84.85% accuracy with only ~174,000 parameters (significantly smaller than baselines like ResNet-20) in a 3-hour search. It also demonstrates flexibility on credit card fraud detection, directly optimizing F1-Score on highly imbalanced tabular data, reaching an F1-Score of 0.71 with a compact ~4,600-parameter network. Paper by Romain Amigon, arXiv:2607.11826.

Paper Overview

Field: Machine Learning Author: Romain Amigon Published: 2026-07-13 arXiv: 2607.11826

Abstract

Neural Architecture Search (NAS) has automated the design of deep learning models but traditionally requires massive computational resources, often measured in thousands of GPU-days. In this paper, we propose a frugal and memetic NAS framework designed to democratize architecture design on consumer-grade hardware.

Our approach combines the global macro-search capabilities of an autoregressive Transformer controller, trained via Reinforcement Learning (RL), with the local micro-exploitation of an Artificial Bee Colony (ABC) algorithm. To prevent premature convergence during the RL phase, we introduce a dynamic entropy mechanism that forces topological exploration upon detection of performance stagnation.

Key Results

Evaluated on a standard GPU (NVIDIA RTX 3060), the hybrid method effectively resolves the 'cold-start' problem inherent to metaheuristics.
By penalizing network depth, the framework actively mitigates model bloat: on CIFAR-10, it discovers an efficient architecture achieving 84.85% accuracy with ~174,000 parameters (significantly smaller than standard baselines such as ResNet-20), with a search time of only 3 hours.
The framework demonstrates flexibility when applied to credit card fraud detection, directly optimizing F1-Score on highly imbalanced tabular data and achieving an F1-Score of 0.71 with a compact network of ~4,600 parameters.