Editing Transformer Architecture (section)

== <span style="color: #FFFFFF;">Applying</span> ==
'''Computing self-attention from scratch in PyTorch:'''

<syntaxhighlight lang="python">
import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Q, K, V: shape (batch, heads, seq_len, d_k)
    """
    d_k = Q.size(-1)
    # Compute attention scores
    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)

    # Apply causal mask (for decoder)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))

    # Softmax to get attention weights
    attn_weights = F.softmax(scores, dim=-1)

    # Weighted sum of values
    return torch.matmul(attn_weights, V), attn_weights

# Example dimensions
batch, heads, seq_len, d_k = 2, 8, 512, 64
Q = torch.randn(batch, heads, seq_len, d_k)
K = torch.randn(batch, heads, seq_len, d_k)
V = torch.randn(batch, heads, seq_len, d_k)

output, weights = scaled_dot_product_attention(Q, K, V)
print(output.shape)  # torch.Size([2, 8, 512, 64])
</syntaxhighlight>

; Key transformer variants and their use cases
: '''BERT''' → Sentence classification, NER, question answering (fine-tuning on labeled data)
: '''GPT-2/3/4''' → Text generation, few-shot learning, instruction following
: '''T5''' → Any-to-any text tasks framed as "text to text"
: '''LLaMA / Mistral''' → Open-weight generation models for local deployment
: '''ViT (Vision Transformer)''' → Image classification by treating patches as tokens
: '''Whisper''' → Speech-to-text using encoder transformer on mel spectrograms
</div>

<div style="background-color: #8B4500; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">