Editing Mixture Of Experts (section)

== <span style="color: #FFFFFF;">Remembering</span> ==
* '''Expert''' — A sub-network (typically an FFN block) that specializes in certain types of inputs.
* '''Gating network (router)''' — A learned function mapping each token to a probability distribution over experts.
* '''Sparse MoE''' — Only top-K experts (K=1 or 2) activated per token; compute stays constant.
* '''Top-K routing''' — Selecting K highest-probability experts per token.
* '''Expert capacity''' — Maximum tokens an expert processes per batch; overflow tokens are dropped.
* '''Load balancing loss''' — Auxiliary loss encouraging uniform token distribution across experts.
* '''Expert collapse''' — Failure mode where router learns to route all tokens to one expert.
* '''Active parameters''' — Parameters actually used per forward pass; far fewer than total in MoE.
* '''Mixtral 8x7B''' — 8 experts per MoE layer, top-2 routing, 47B total, ~13B active per token.
* '''Switch Transformer''' — Google's top-1 routing MoE scaled to 1.6 trillion parameters.
* '''Expert parallelism''' — Distributing different experts across different GPUs for scale.
</div>

<div style="background-color: #006400; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">