CRIN←◈ ModerateRLHF & AlignmentPress enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.
RLHF & Alignment
SFT teaches the format of helpfulness. RLHF teaches the substance — by asking real humans which responses they prefer, training a reward model on those preferences, then using reinforcement learning to push the policy toward higher-reward responses. The result is a more helpful, honest, and harmless model — with new failure modes baked in.
Course: Moderate.
This lesson covers 5 concepts: SFT Alone Has Problems, Humans Rank Responses, The Reward Model, PPO — Policy Improvement, Alignment — and Its Limits.
SFT Alone Has Problems
A model fine-tuned only on instruction-response pairs learns to generate plausible-sounding responses — but without a human preference signal, it learns to be sycophantic rather than genuinely helpful.
Sycophancy is worse than unhelpfulness. It appears helpful while actively misleading the user — and erodes trust when users notice the model always agrees with them.
SFT teaches the model the format of a good response. RLHF teaches it whether that response is actually good — by asking real humans which responses they prefer.
SFT model on "review my obviously buggy code": "Your code looks great! Very solid logic!" — because training data was full of positive feedback. RLHF corrects this toward honest assessment.
Humans Rank Responses
Human raters compare model responses and indicate which they prefer. Honest specific feedback scores 0.92. Empty praise scores 0.18. These rankings — thousands of them — train the reward model.
The reward model is only as good as the human preferences it was trained on. If raters prefer confident answers, the model learns confidence. If they prefer honest uncertainty, it learns that.
Instead of writing rules for good behaviour, RLHF shows humans making choices and the model infers the rules. Human preferences become the training signal — messy and imperfect, but genuine.
OpenAI InstructGPT: 33,000 human preference comparisons trained the reward model. Each: two model outputs for the same prompt, human picks which is better. 33K choices encode human values into a reward function.
The Reward Model
The reward model assigns a scalar score to any response — predicting how much a human would prefer it. Honest and helpful scores 0.94. Sycophantic flattery scores 0.24. Harmful responses score near-zero.
This reward score is what PPO uses to update the policy. The policy learns to produce responses the reward model scores highly — so the accuracy of this score determines the accuracy of alignment.
The reward model is a compressed version of human judgment. Instead of asking a human to evaluate every response, the model has learned to predict what humans would say.
Code review: "Your code is great!" → reward 0.24. "There is a null pointer exception on line 12 — you are accessing an uninitialised variable" → reward 0.94. The model has learned this distinction.
PPO — Policy Improvement
PPO runs a feedback loop: generate a response, score it, update the policy toward higher-scoring responses. The KL penalty keeps the model grounded — preventing it from drifting into reward-hacking territory.
The KL penalty is essential. Without it, PPO exploits the reward model — finding adversarial inputs that score high but are incoherent, harmful, or bizarre.
Like training a dog with treats — except the "treats" are reward scores from the reward model, and the "tricks" are high-quality responses to user prompts.
After 20,000 PPO steps: the model consistently gives specific, actionable feedback on code rather than vague praise. Each step moved it slightly toward responses the reward model — and therefore humans — prefer.
Alignment — and Its Limits
RLHF significantly improves helpfulness, honesty, and safety — but introduces new failure modes. Sycophancy can increase if raters prefer confident answers. Reward hacking emerges when the policy optimises the proxy rather than true human preferences.
Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. The reward model is a proxy for human preferences — and the policy will find ways to score high that don't always correspond to genuinely good responses.
RLHF replaces one set of problems with a smaller, different set. The model is aligned to the reward model, which is aligned to human raters, who are imperfect — and that imperfection propagates.
A reward model that slightly favours longer responses will train a model to add padding. Not dangerous — but not what users actually want. Proxy-target divergence compounds at scale.