Editing Rlhf (section)

== <span style="color: #FFFFFF;">Analyzing</span> ==
{| class="wikitable"
|+ RLHF Method Comparison
! Method !! Complexity !! Human Labels Needed !! Reward Hacking Risk !! Open Source Support
|-
| PPO-RLHF || Very high || Many (comparisons) || Medium || TRL, OpenRLHF
|-
| DPO || Low || Many (comparisons) || Low || TRL, Axolotl
|-
| RLAIF || Medium || Few (AI generates) || Medium || Custom
|-
| Constitutional AI || Medium || Few || Low || Custom
|-
| SFT only || Very low || Demonstrations || None || Transformers
|}

'''Failure modes''': Sycophancy — model tells users what they want to hear. Verbosity bias — human raters often prefer longer responses even when shorter is better; model learns to pad. Refusal overuse — safety-trained models refuse benign requests. Distribution shift between raters and deployment users causes misaligned behavior. Reward model collapse on out-of-distribution prompts.
</div>

<div style="background-color: #483D8B; color: #FFFFFF; padding: 20px; border-radius: 8px; margin-bottom: 15px;">