Editing Neural Architecture Search (section)

== <span style="color: #FFFFFF;">Understanding</span> ==
NAS frames architecture design as an optimization problem: find the architecture A* = argmax''{A ∈ search''space} Performance(A, task). The three key components are what to search (search space), how to search (search strategy), and how to evaluate candidates (performance estimation).

'''Early NAS''' (Zoph & Le, 2017): Used an RNN controller (meta-learner) to generate architecture descriptions. Each generated architecture was trained to convergence on CIFAR-10, and its accuracy was used as a reward signal to update the controller via REINFORCE. This worked but required 800 GPUs for 28 days — prohibitively expensive.

'''DARTS''' (Liu et al., 2019): Made NAS practical by making architecture selection differentiable. Instead of discrete choices, DARTS maintains a continuous mixture: output = Σ''o α''o · op''o(x), where α''o are learnable architecture parameters. Gradient descent simultaneously optimizes network weights (on training data) and architecture parameters (on validation data). At the end, the highest-weight operation at each edge is selected. This reduced search to a few GPU-days.

'''One-shot NAS / Weight sharing''': Train a single supernet where all operations share weights. Individual sub-architectures inherit weights from the supernet for fast evaluation. Hardware-aware NAS adds latency or FLOPs as a penalty, enabling automatic design for mobile, edge, or server targets.

'''The proxy problem''': Architectures found by searching on CIFAR-10 may not transfer to ImageNet or other tasks. Hardware benchmarks in simulation may differ from actual device latency. Expert NAS practitioners search directly on target tasks and hardware when feasible.
</div>

<div style="background-color: #8B0000; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">