Editing Transfer Learning (section)

== <span style="color: #FFFFFF;">Creating</span> ==
Designing a transfer learning pipeline for a new domain:

'''1. Source model selection'''
<syntaxhighlight lang="text">
Task modality:
├── Images → ImageNet pre-trained (ResNet, EfficientNet, ViT)
│   ├── Medical images → MedCLIP, RadDINO, BioViL
│   ├── Satellite images → SatMAE, Prithvi
│   └── Microscopy → CellViT, BioCLIP
├── Text → BERT, RoBERTa, DeBERTa (encoder); LLaMA (decoder)
│   ├── Scientific text → SciBERT, BioMedBERT
│   ├── Legal text → LegalBERT
│   └── Code → CodeBERT, StarCoder
└── Multimodal → CLIP, Flamingo, LLaVA
</syntaxhighlight>

'''2. Progressive fine-tuning schedule'''
<syntaxhighlight lang="text">
[Optional] Domain-adaptive pre-training (DAP):
  - Pre-train on in-domain unlabeled text/images (MLM or MAE)
  - No labels required; leverages domain corpus
    ↓
Phase 1: Feature extraction (head only, 5–10 epochs)
  - Verify features are useful for target task
  - Establishes good head initialization
    ↓
Phase 2: Gradual unfreezing
  - Unfreeze last block → train 5 epochs
  - Unfreeze next block → train 5 epochs
  - Continue until performance plateaus
    ↓
Phase 3: Full fine-tuning (optional, if data permits)
  - Very low learning rate (1e-5)
  - Early stopping on validation metric
</syntaxhighlight>

'''3. Preventing catastrophic forgetting'''
* Elastic Weight Consolidation (EWC): penalize updates to weights important for source task
* Learning Without Forgetting (LwF): use original model as soft-label teacher
* For LLMs: KL penalty on fine-tuned vs. original model's output distribution (the RLHF approach)
* Replay: mix small fraction of source-domain data into fine-tuning batches

[[Category:Artificial Intelligence]]
[[Category:Machine Learning]]
[[Category:Deep Learning]]
</div>