Editing Reinforcement Learning from Human Feedback (section)

== <span style="color: #FFFFFF;">Applying</span> ==
'''DPO fine-tuning with TRL:'''
<syntaxhighlight lang="python">
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig

# Load base model + tokenizer
model_name = "meta-llama/Llama-3.2-1B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name)
ref_model = AutoModelForCausalLM.from_pretrained(model_name)  # Frozen reference
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Preference dataset format: {prompt, chosen, rejected}
# "chosen" = preferred response, "rejected" = less preferred
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
# Dataset format:
# {"prompt": "What is...", "chosen": "Great answer...", "rejected": "Bad answer..."}

# DPO training config
config = DPOConfig(
    beta=0.1,                  # KL penalty weight (higher = stay closer to reference)
    learning_rate=5e-7,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    max_length=1024,
    max_prompt_length=512,
    output_dir="./dpo_output",
)

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,       # Reference model for KL constraint
    args=config,
    train_dataset=dataset,
    tokenizer=tokenizer,
)
trainer.train()
trainer.save_model("./aligned_model")
</syntaxhighlight>

; RLHF variant comparison
: '''Original RLHF (PPO)''' → Best-studied; used in ChatGPT, Claude 1; complex to implement
: '''DPO''' → Simpler; no explicit reward model; now dominant in open-source
: '''RLAIF''' → AI-generated feedback at scale; reduces human annotation cost
: '''Constitutional AI''' → Rule-based self-critique; Anthropic's approach for safety
: '''GRPO (Group Relative Policy Optimization)''' → DeepSeek's variant; no value model needed
: '''Best-of-N''' → Simple baseline; generate N samples, pick highest reward model score
</div>

<div style="background-color: #8B4500; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">