Editing Scaling Laws (section)

== <span style="color: #FFFFFF;">Remembering</span> ==
* '''Scaling law''' — A power-law relationship between model performance and compute, parameters, or data.
* '''Compute (C)''' — Total floating point operations used in training, measured in FLOPs.
* '''Parameters (N)''' — The total number of learnable weights in the model.
* '''Tokens (D)''' — The number of training data tokens the model was trained on.
* '''Cross-entropy loss''' — The primary metric in scaling law studies; lower = better language modeling.
* '''Kaplan scaling laws''' — OpenAI's 2020 paper showing loss scales as a power law with N, D, C.
* '''Chinchilla scaling laws''' — DeepMind's 2022 finding that optimal training allocates equal compute to model size and training data (N ∝ D).
* '''Compute-optimal model''' — A model trained with the optimal N and D for a given compute budget, per Chinchilla.
* '''Emergent ability''' — A capability that appears only at certain model scale, not visible in smaller models.
* '''Phase transition''' — Abrupt, discontinuous improvement in a capability as scale increases.
* '''Irreducible loss''' — The minimum achievable loss on the data distribution; sets a floor on scaling improvements.
* '''Inference scaling''' — Using additional compute at inference time (more tokens of reasoning, chain-of-thought, search) to improve outputs.
* '''Test-time compute''' — Compute spent during inference to improve answer quality (e.g., best-of-N sampling, process reward models).
</div>

<div style="background-color: #006400; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">