Lesson 102: RLHF & DPO — Instruction Fine-Tuning

Discussion prompt

Before we open Lesson 102: RLHF & DPO — Instruction Fine-Tuning: without looking back, what was the main idea of LoRA — Low-Rank Adaptation, and what could you do by the end of it that you could not do before?

Hint: One sentence for the idea, one for the skill. If the second one is blank, that is the part to revisit.

Comparison matrix

From Pretraining vs instruction following: refill the data column from what you know. The rest of the table is as it appeared.

Discussion prompt

Each stage feeds the next: SFT initializes the policy and the RM; the RM provides the reward signal; PPO then optimizes the policy while the SFT model acts as a frozen reference for the KL term.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Predict first

The table runs: 1 (init) | 2.7608 · 2 | 2.6412

In SFT: cross-entropy on the response only, given the rows so far: what is the next one — the row where step is 3?

Correct: 3 | 2.5241

Why: The relationship between the columns, not the individual numbers, is what generates the next row. We do NOT train on the prompt tokens; supervising on them teaches the model to copy prompts, not respond to them.

Comparison matrix

From SFT: cross-entropy on the response only: every row here is a choice with a cost. Fill the loss column, then say which row you would actually pick and what you give up for it.

Discussion prompt

Explain Preference data and the Bradley-Terry model by analogy to something with no Machine Learning in it at all — a queue, a recipe, a map, a bank balance, whatever fits. Then say where your analogy breaks.

Hint: An analogy that never breaks is not an analogy, it is the same idea wearing a hat. Find the seam — that is the part that is actually new.

Fill in the blanks

From Reward model: pairwise loss computation — one line has had its right-hand side removed. Put it back.

class ToyRewardModel(nn.Module):
def __init__(self, vocab, d):
super().__init__()
self.embed = nn.Embedding(vocab, d)
self.head = nn.Linear(d, 1)
def forward(self, x):
return self.head(self.embed(x).mean(dim=1)).squeeze(-1)

rm = ToyRewardModel(vocab=8, d=16)
chosen = torch.tensor([[1, 3, 5, 6, 7]]) # higher quality
rejected = torch.tensor([[1, 2, 2, 2, 2]]) # lower quality
r_w = rm(chosen); r_l = rm(rejected)
loss_rm = -F.logsigmoid(r_w - r_l).mean()
print(f'r(chosen)=___ r(rejected)=___')
print(f'RM loss (Bradley-Terry)=___')

Why: chosen is what everything below it consumes, so the wrong expression here fails later and somewhere else. The RM architecture is the same as the SFT model but with a scalar head replacing the vocab projection; it can be initialized from the SFT checkpoint.

Annotate

Walk the callouts on Reward model: pairwise loss computation. Each one is a place this is easy to get subtly wrong.

• The RM architecture is the same as the SFT model but with a scalar head replacing the vocab projection; it can be initialized from the SFT checkpoint.
• After training, the RM loss minimizes when r(y_w) >> r(y_l) for all preference pairs — the model has learned the reward signal from human labels.

Predict first

A student writes this, and it looks reasonable:

Annotators assign numeric scores (1–10) to each completion separately. Train the RM with MSE: loss = MSE(r_phi(x,y), score).

It is wrong. Say what breaks — and say it before you turn the page.

Correct: Seems natural — match the model's scalar to the annotator's rating.

Collect pairwise comparisons (A vs B, which is better?). Train the RM with Bradley-Terry loss on the margin: loss = -log_sigmoid(r_w - r_l).

Why: Seems natural — match the model's scalar to the annotator's rating.

Discussion prompt

The rule this trap just fixed:

Relative comparison is far more consistent for humans than absolute scoring; the model learns what relatively better means, not a calibrated scale.

Now break it on purpose. Build a case that violates it and follow the consequences until something visibly fails. Where does the failure first show up — and would you have noticed it if you had not been looking?

Hint: The dangerous rules are the ones whose violation still produces an answer. If yours fails loudly, try to find one that fails quietly.

Discussion prompt

After SFT and RM training, the policy π_θ (initialized from π_SFT) is optimized to maximize expected reward while staying close to the SFT reference distribution.

That is stated as though it always holds. Do one of two things: produce a case where it fails, or say precisely what rules such a case out. "It just does" is not on the menu.

Hint: Hunt at the extremes first — zero, one, negative, empty, equal. If every extreme survives, the reason they survive is the proof.

Fill in the blanks

From PPO shaped reward: r_env − β·KL — one line has had its right-hand side removed. Put it back.

torch.manual_seed(7)
pol_logits = torch.randn(6, 8) # policy (6 tokens, vocab=8)
ref_logits = torch.randn(6, 8) # frozen SFT reference
pol_lp = F.log_softmax(pol_logits, dim=-1)
ref_lp = F.log_softmax(ref_logits, dim=-1)
# KL per token: sum over vocab
kl = (pol_lp.exp() * (pol_lp - ref_lp)).sum(dim=-1)
print('KL per token:', kl.detach().numpy().round(4))
print('Mean KL:', round(kl.mean().item(), 4))
r_env = 0.85 # reward from trained RM
beta = 0.1
shaped = r_env - beta * kl.mean().item()
print('shaped reward:', round(shaped, 4))

Why: ref_lp is what everything below it consumes, so the wrong expression here fails later and somewhere else. KL(pi || pi_ref) = sum_v pi(v|x) * (log pi(v|x) - log pi_ref(v|x)) measures how far the policy has drifted; summed/averaged over the response tokens.

Comparison matrix

From PPO shaped reward: r_env − β·KL: refill the KL(pi||pi_ref) column from what you know. The rest of the table is as it appeared.

Sort into buckets

These are the pieces of Lesson 102: RLHF & DPO — Instruction Fine-Tuning, out of order. Put each one back under the part of the lesson it belongs to.

Why raw pretraining is not enough

Pretraining vs instruction following; The 3-step RLHF pipeline

Step 2: Reward Model from Human Preferences

Preference data and the Bradley-Terry model; Reward model: pairwise loss computation

Step 3: PPO with KL Penalty

The policy optimization objective in RLHF; PPO shaped reward: r_env − β·KL; Why PPO — not vanilla policy gradient?

s1

Why raw pretraining is not enough is where Lesson 102: RLHF & DPO — Instruction Fine-Tuning puts Pretraining vs instruction following, The 3-step RLHF pipeline. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s2

Step 2: Reward Model from Human Preferences is where Lesson 102: RLHF & DPO — Instruction Fine-Tuning puts Preference data and the Bradley-Terry model, Reward model: pairwise loss computation. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

s3

Step 3: PPO with KL Penalty is where Lesson 102: RLHF & DPO — Instruction Fine-Tuning puts The policy optimization objective in RLHF, PPO shaped reward: r_env − β·KL, Why PPO — not vanilla policy gradient?. Knowing which part of the lesson a problem belongs to is most of knowing which method to reach for.

Predict first

The table runs: log pi(y_w|x) - log pi_ref(y_w|x) | 0.6000 | -4.2 - (-4.8) · log pi(y_l|x) - log pi_ref(y_l|x) | -0.2000 | -5.1 - (-4.9) · imp_w = beta * 0.6 | 0.0600 | implicit reward for chosen · imp_l = beta * (-0.2) | -0.0200 | implicit reward for rejected · margin = imp_w - imp_l | 0.0800 | beta * (ratio_w - ratio_l)

In DPO loss: computing implicit rewards and margin, given the rows so far: what is the next one — the row where quantity is DPO loss = -log sigma(0.08)?

Correct: DPO loss = -log sigma(0.08) | 0.6539 | positive: policy not yet trained

Why: The relationship between the columns, not the individual numbers, is what generates the next row. Sum the per-token log-probs under both the policy and the reference, then take the difference.

Comparison matrix

From DPO loss: computing implicit rewards and margin: every row here is a choice with a cost. Fill the value column, then say which row you would actually pick and what you give up for it.

Predict first

A student writes this, and it looks reasonable:

After training with DPO, keep both pi_theta and pi_ref loaded at inference time to compute the implicit reward before each generation.

It is wrong. Say what breaks — and say it before you turn the page.

Correct: The DPO objective is framed around the ratio, so the reference must stay active.

The reference model pi_ref is needed only during DPO training to compute the implicit reward ratios. At inference, just sample from pi_theta normally.

Why: The DPO objective is framed around the ratio, so the reference must stay active.

Sort into buckets

Some of these hold up and some are the exact mistakes this lesson is built to prevent. Sort them.

Holds up

Annotators see a prompt x and two completions y_w (winner) and y_l (loser) and mark which they prefer. The reward model learns to assign higher scalar reward to y_w.; After SFT and RM training, the policy π_θ (initialized from π_SFT) is optimized to maximize expected reward while staying close to the SFT reference distribution.; Vanilla REINFORCE has high variance and can take catastrophically large gradient steps. PPO clips the probability ratio to a trust region, preventing destructive updates.

Breaks

Annotators assign numeric scores (1–10) to each completion separately. Train the RM with MSE: loss = MSE(r_phi(x,y), score).; After training with DPO, keep both pi_theta and pi_ref loaded at inference time to compute the implicit reward before each generation.

sound

These are stated as this lesson states them — each one survives the edge cases Lesson 102: RLHF & DPO — Instruction Fine-Tuning puts it through.

flawed

Each of these is lifted from a trap in this deck: reasonable-sounding, and wrong in a way that only shows up once you rely on it.

Comparison matrix

From RLHF vs DPO: trade-offs: refill the RLHF (PPO) column from what you know. The rest of the table is as it appeared.

Discussion prompt

Run Recipe: RLHF or DPO on a new task with this step confiscated:

RLHF path: Train RM with Bradley-Terry loss: L = -log σ(r(x,y_w) − r(x,y_l)). Then run PPO maximizing E[r(x,y) − β·KL(π||π_SFT)], loading a value network as the PPO critic.

Is it still possible? If it is, say what takes its place and what it costs you. If it is not, say exactly what that step was providing that nothing else does.

Hint: A step you can drop for free was never load-bearing. If you cannot drop it, name the thing that goes wrong the moment it is gone.

Discussion prompt

Recipe: RLHF or DPO on a new task works on the cases you have just seen. Push it to the edge: what is the most degenerate input it still handles — empty, zero, one item, everything equal — and what is the first case where it stops being true? Name the case, not just "it breaks".

Hint: Try the smallest legal input, then the largest, then the one where two things collide. Methods are specified at their edges; the middle takes care of itself.

Eliminate the wrong options

In the standard 3-step RLHF pipeline, which model is used as the frozen reference during the PPO stage?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. The reward model r_phi
• B. The original pretrained LM before any fine-tuning
• C. The SFT model pi_SFT
• D. The value network (PPO critic)

Survives elimination: C

Why: The SFT model pi_SFT is frozen and used to compute the KL penalty KL(pi_theta || pi_SFT) during PPO. It anchors the policy near the instruction-following baseline, preventing reward hacking.

Check your understanding

In the standard 3-step RLHF pipeline, which model is used as the frozen reference during the PPO stage?

• A. The reward model r_phi
• B. The original pretrained LM before any fine-tuning
• C. The SFT model pi_SFT (correct)
• D. The value network (PPO critic)

Answer: C

Why: The SFT model pi_SFT is frozen and used to compute the KL penalty KL(pi_theta || pi_SFT) during PPO. It anchors the policy near the instruction-following baseline, preventing reward hacking.

Why A tempts people

The reward model r_phi provides the environment reward signal, but it is not the KL reference — it is a separate frozen model queried for scalar scores.

Why B tempts people

The raw pretrained LM is not used after SFT. Using it as the reference would allow the policy to drift far from instruction-following behavior before the KL kicks in.

Why D tempts people

The value network (critic) estimates expected future reward for PPO's advantage computation — it is not the reference distribution for the KL divergence term.

Predict first

A reward model outputs r(x, y_w) = 1.2 and r(x, y_l) = 0.7 for a preference pair. What is the Bradley-Terry loss for this example (to 3 decimal places)?

Answer it in your own words, now, with nothing to choose from. The options are on the next slide — and picking the right one off a list is an easier skill than producing it.

Correct: -log σ(0.5) ≈ 0.474

Why: margin = r(y_w) - r(y_l) = 1.2 - 0.7 = 0.5. Loss = -log σ(0.5) = -log(1/(1+e^{-0.5})) ≈ 0.474. The model is partially correct — loss is positive and would decrease as the margin grows.

Check your understanding

A reward model outputs r(x, y_w) = 1.2 and r(x, y_l) = 0.7 for a preference pair. What is the Bradley-Terry loss for this example (to 3 decimal places)?

• A. -log σ(0.5) ≈ 0.474 (correct)
• B. -log σ(1.9) ≈ 0.135
• C. log σ(0.5) ≈ -0.474
• D. -log σ(-0.5) ≈ 0.974

Answer: A

Why: margin = r(y_w) - r(y_l) = 1.2 - 0.7 = 0.5. Loss = -log σ(0.5) = -log(1/(1+e^{-0.5})) ≈ 0.474. The model is partially correct — loss is positive and would decrease as the margin grows.

Why B tempts people

The margin 1.9 would arise if both scores were summed (1.2 + 0.7) rather than subtracted. The Bradley-Terry model uses the difference, not the sum.

Why C tempts people

The sign is wrong — the RM loss is the negative log-sigmoid of the margin, making it a positive quantity that gradient descent minimizes.

Why D tempts people

A margin of -0.5 would mean r(y_l) > r(y_w), which is not the case here. This would correspond to the RM ranking the completions backwards.

Eliminate the wrong options

With beta=0.1, a trained policy has log pi_theta(y_w|x) - log pi_ref(y_w|x) = 3.0 and log pi_theta(y_l|x) - log pi_ref(y_l|x) = -1.0. Which statement is correct?

3 of these 4 are wrong. Strike them one at a time, and say what rules each one out before you strike the next. The survivor is the answer.

• A. DPO loss margin = beta*(3.0 - (-1.0)) = 0.4; the policy correctly ranks y_w above y_l
• B. DPO loss margin = 3.0 - (-1.0) = 4.0; beta is applied only to the chosen completion
• C. DPO loss margin = beta*(3.0 + (-1.0)) = 0.2; the two log-ratios are summed
• D. DPO loss margin = beta*(3.0 - (-1.0)) = 0.4; but the policy still minimizes loss when y_l is ranked higher

Survives elimination: A

Why: margin = beta(ratio_w - ratio_l) = 0.1(3.0 - (-1.0)) = 0.4. A positive margin means sigma(0.4) > 0.5, so -log sigma(0.4) < log 2 ≈ 0.693 — the policy is correctly placing y_w above y_l and the loss is below the random-init baseline.

Check your understanding

With beta=0.1, a trained policy has log pi_theta(y_w|x) - log pi_ref(y_w|x) = 3.0 and log pi_theta(y_l|x) - log pi_ref(y_l|x) = -1.0. Which statement is correct?

• A. DPO loss margin = beta*(3.0 - (-1.0)) = 0.4; the policy correctly ranks y_w above y_l (correct)
• B. DPO loss margin = 3.0 - (-1.0) = 4.0; beta is applied only to the chosen completion
• C. DPO loss margin = beta*(3.0 + (-1.0)) = 0.2; the two log-ratios are summed
• D. DPO loss margin = beta*(3.0 - (-1.0)) = 0.4; but the policy still minimizes loss when y_l is ranked higher

Answer: A

Why: margin = beta(ratio_w - ratio_l) = 0.1(3.0 - (-1.0)) = 0.4. A positive margin means sigma(0.4) > 0.5, so -log sigma(0.4) < log 2 ≈ 0.693 — the policy is correctly placing y_w above y_l and the loss is below the random-init baseline.

Why B tempts people

beta multiplies the difference of both log-ratios, not just the chosen one. Dropping beta inflates the margin by 10x and gives the wrong scale.

Why C tempts people

The DPO formula subtracts the rejected ratio from the chosen ratio: (ratio_w - ratio_l). Summing them (3.0 + (-1.0) = 2.0) confuses the sign of the rejected term.

Why D tempts people

A positive margin means the policy already ranks y_w above y_l (imp_w = 0.3 > imp_l = -0.1). Loss is minimized when the margin is large and positive, not when y_l is ranked higher.

Draw it

One page, no notation unless you need it: draw how these connect — Why raw pretraining is not enough · Step 1: Supervised Fine-Tuning (SFT) · Step 2: Reward Model from Human Preferences · Step 3: PPO with KL Penalty · DPO: Direct Preference Optimization. Put an arrow wherever one of them is what makes another possible, and label the arrow with why.