neural architecture search
Neural Architecture Search (NAS) — using algorithms to automatically discover optimal neural network architectures instead of relying on human design, a key branch of AutoML.
The Problem
- Architecture design is manual and requires expert intuition
- Huge design space: Number of layers, filter sizes, connections, attention heads, activation functions
- Humans can't explore all possibilities
Search Strategies
- Reinforcement Learning NAS: A controller network proposes architectures; reward = validation accuracy. Original method (Google, 2017). Cost: 800 GPU-days
- Evolutionary NAS: Mutate and evolve a population of architectures. Similar cost to RL approach
- Differentiable NAS (DARTS): Make architecture choices continuous and differentiable → use gradient descent to search. Cost: 1-4 GPU-days (1000x cheaper)
- One-Shot NAS: Train a single supernet containing all candidate architectures, then extract the best subnet
Notable Results
- NASNet: Found architectures better than human-designed ResNet
- EfficientNet: NAS-designed CNN that set ImageNet records
- MnasNet: NAS for mobile — Pareto-optimal speed vs accuracy
Limitations
- Search space must be carefully defined by humans
- Results often aren't dramatically better than well-designed manual architectures
- Reproducibility challenges
NAS demonstrated that machines can design neural networks — but the community has shifted toward scaling known architectures rather than searching for new ones.