Editing Transformer Architecture (section)

== <span style="color: #FFFFFF;">Creating</span> ==
Designing a transformer-based system architecture:

'''1. Choose the right variant for the task'''
<syntaxhighlight lang="text">
Task Classification:
├── Generating text/code → Decoder-only (GPT/LLaMA family)
├── Encoding for search/classification → Encoder-only (BERT/RoBERTa/E5)
├── Translation/Summarization → Encoder-Decoder (T5/BART)
└── Images → ViT or CLIP (vision-language)
</syntaxhighlight>

'''2. Efficient attention for long contexts'''
* Flash Attention 2 for memory-efficient exact attention
* Grouped Query Attention (GQA) — reduces KV-cache by sharing keys/values across query heads
* Sliding window attention for documents

'''3. Production serving architecture'''
<syntaxhighlight lang="text">
User Request
    ↓
Load Balancer
    ↓
[Tokenizer service]
    ↓
[Inference cluster: vLLM / TGI / Triton]
    ↓ (continuous batching, PagedAttention)
[GPU cluster with tensor parallelism]
    ↓
[Detokenizer + safety filter]
    ↓
Response
</syntaxhighlight>

'''4. Key efficiency techniques'''
* Quantization: INT8/INT4 weights reduce memory 2–4× with minimal quality loss
* Speculative decoding: a small "draft" model proposes tokens, a large model verifies them in parallel — 2–3× throughput
* Prefix caching: cache shared system prompt KV-states across requests
* Batching: group requests sharing prefixes to maximize GPU utilization

[[Category:Artificial Intelligence]]
[[Category:Deep Learning]]
[[Category:Natural Language Processing]]
[[Category:Transformer Architecture]]
</div>