Editing Transformer Architecture (section)

== <span style="color: #FFFFFF;">Analyzing</span> ==
{| class="wikitable"
|+ Transformer Architecture Trade-offs
! Aspect !! Benefit !! Limitation
|-
| Self-attention || Captures long-range dependencies in O(1) steps || O(n²) memory and compute with sequence length
|-
| Parallelism || Trains much faster than RNNs on modern hardware || Requires large GPU memory for long contexts
|-
| Scale || Performance consistently improves with more parameters and data || Training cost is enormous (millions of dollars for frontier models)
|-
| Context window || Modern models handle 100k+ tokens || KV-cache memory grows linearly; long-context retrieval degrades
|-
| Positional encoding || Sinusoidal or RoPE encodes position || Generalization beyond training context length is degraded
|}

'''Failure modes and nuances:'''
* '''Attention sink''' — Research shows early tokens receive disproportionate attention regardless of relevance, a consequence of the softmax function needing to sum to 1.
* '''Quadratic scaling''' — Self-attention complexity is O(n²) in sequence length, making very long documents expensive. Mitigations: Flash Attention, sliding window attention (Longformer), linear attention approximations.
* '''Position generalization''' — Transformers trained with absolute positional encodings often fail on sequences longer than seen in training. RoPE (Rotary Position Embeddings) and ALiBi improve this.
* '''Repetition''' — Decoder-only models can fall into repetitive loops; nucleus sampling (top-p) and repetition penalties help.
</div>

<div style="background-color: #483D8B; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">