RLHF & Alignment
SFT teaches the format of helpfulness. RLHF teaches the substance — by asking real humans which responses they prefer, training a reward model on those preferences, then using reinforcement learning to push the policy toward higher-reward responses. The result is a more helpful, honest, and harmless model — with new failure modes baked in.
Polite and useless
Polite, confident, and no help at all.
Fine-tuning taught it to answer. It didn’t teach it what makes an answer good.
People pick one
Show people two answers to the same question. Ask which one is better.
Collect a great many of these choices and you have a record of what people prefer.
Rate the answers
Where would most people put each answer?
A judge model
Those choices train a second model to score any answer the way people would.
It’s called a reward model. It can score far more answers than people ever could read.
Practice for points
The chatbot writes answers, the judge scores them, and it’s nudged toward higher scores.
Learning by trying and being scored, not by copying examples, is called reinforcement learning.
Learning to flatter
Suppose people slightly prefer upbeat replies. Which word would the model learn to start with?
Its reply begins “Your code…”
Back to the review
Labs choose raters and instructions carefully, so honest, specific answers win the comparisons.
Done well, the same review gets a real answer.
Trained on preferences
People’s choices train a judge, and the model practises to please it, flaws included.
This is reinforcement learning from human feedback. Shaping a model toward what people want is alignment.