Editing Neural Architecture Search (section)

== <span style="color: #FFFFFF;">Applying</span> ==
'''Architecture search with DARTS (simplified):'''
<syntaxhighlight lang="python">
import torch
import torch.nn as nn
import torch.nn.functional as F

OPERATIONS = {
    'skip_connect': lambda C: nn.Identity(),
    'sep_conv_3x3': lambda C: SepConv(C, C, 3, 1),
    'dil_conv_5x5': lambda C: DilConv(C, C, 5, 2, 2),
    'max_pool_3x3': lambda C: nn.MaxPool2d(3, 1, 1),
    'avg_pool_3x3': lambda C: nn.AvgPool2d(3, 1, 1, count_include_pad=False),
}

class MixedOp(nn.Module):
    """A mixed operation where each candidate is weighted by architecture parameter."""
    def __init__(self, C):
        super().__init__()
        self.ops = nn.ModuleList([op(C) for op in OPERATIONS.values()])

    def forward(self, x, weights):
        # weights: softmax of architecture parameters α
        return sum(w * op(x) for w, op in zip(weights, self.ops))

# Architecture parameters (separate from network weights)
arch_params = nn.Parameter(torch.randn(n_edges, len(OPERATIONS)))

# Bi-level optimization:
# 1. Update network weights on train set (standard gradient)
# 2. Update arch_params on val set (gradient through mixed ops)
# Final: discretize by argmax(softmax(arch_params)) per edge
</syntaxhighlight>

; NAS approach selection
: '''Compute budget <1 GPU-day''' → DARTS, GDAS, or random search (surprisingly competitive)
: '''Hardware-specific target''' → ProxylessNAS, SNAS, Once-for-All
: '''Zero search cost''' → Zero-cost proxies (NASWOT, synflow) — correlate with performance without training
: '''Production ML teams''' → AutoML platforms: Google AutoML, Azure AutoML, Auto-Sklearn
</div>

<div style="background-color: #8B4500; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">