[LLM 3/10] RLHF and PPO: Training a Model on a Reward You Cannot Differentiate

In chapter 2 we taught the model by "imitating the answer key" one token at a time. But the qualities that make an AI assistant genuinely usable — correct, polite, not making things up, not sliding into English — have no answer key to imitate, and cannot be written directly as a loss function. This chapter is the most classical answer to that problem: RLHF (Reinforcement Learning from Human Feedback) with PPO. We will train a real reward model from Thai preference pairs, then write the PPO loop ourselves from scratch in about 120 lines, and close with my favorite experiment in the series: taking off the KL leash and watching the model cheat the reward live. This is deliberately the heaviest chapter of the series, because chapter 4 (DPO) and chapter 5 (GRPO) both start from this chapter's equations and each choose a different piece to delete.

Open in Colab03_rlhf_ppo.ipynb

Thai LLM seriesPart 3 of 10

1. 1Continue Pretraining
2. 2SFT and LoRA
3. 3RLHF and PPO(you are here)
4. 4DPO: Direct Preference Optimization
5. 5GRPO
6. 6Context Distillation
7. 7Model Distillation
8. 8Guardrails
9. 9Benchmarking
10. 10Deployment

1. The Problem

SFT in chapter 2 carries one hidden assumption: there must be an answer key to imitate. But think about what we actually want — say, "solve the math problem correctly, and explain it in readable Thai." That sentence has no single answer key: good answers come in a hundred shapes, and "readable" cannot be written as an equation.

Try to optimize these things directly and you always run into two walls:

Wall one — quality cannot be written as a loss. "Better" cannot be defined as a function, but humans are very good at comparing: show them two answers and ask which one they prefer, and they can do it immediately and fairly consistently. So the data you can actually collect comes in threes: a prompt $x$, the chosen answer $y_w$, the rejected answer $y_l$.

Wall two — even with a score, you cannot backprop. Suppose a magical function $r(x,y)$ scored every answer. You still couldn't train supervised, because the answer $y$ comes from sampling tokens one at a time. The score arrives after the sampling is done, and derivatives cannot travel back through sampling — the path from $r$ back to the weights $\theta$ breaks exactly there.

The road you'd like to take	The wall it hits
Write a loss for "a good answer" directly	"Good" has no equation — there are only comparisons
Have humans score, then backprop	The score sits behind token sampling — gradients can't walk through sampling
Have humans score live during training	Humans can't keep pace with even a sliver of the rollouts

Hence the chapter's title: we are about to optimize a reward we cannot differentiate. The tool that can do that is called reinforcement learning.

2. What We're Going to Do

RLHF gets past both walls with a two-step walk:

• Stage A — Reward Model: train a model $r_\phi(x,y)$ to imitate human comparisons from preference pairs (defeats wall one, and stands in for humans who can't score fast enough)
• Stage B — PPO: use policy-gradient RL to push the policy toward $r_\phi$'s scores without differentiating through the sampling (defeats wall two), with a KL leash holding it back from fleeing the starting model

The price you pay is complexity: during training there are four models resident in VRAM at once — the policy $\pi_\theta$ (the one being trained), the reference $\pi_{\text{ref}}$ (the frozen starting model), the reward model $r_\phi$, and a value network $V_\psi$ that hasn't introduced itself yet (wait for section 3.4).

The core idea of this chapter

RLHF is the optimization of a reward you cannot differentiate, through an imperfect proxy (the reward model). And optimizing a proxy hard, with nothing to restrain you, always fails by Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

The KL term in equation 3.2 is therefore not a regularizer sprinkled in for good luck — it is the only thing standing between you and reward hacking. Section 8 will prove that sentence by taking it out before your eyes.

And one more sentence to hold on to for the whole series: this chapter's objective is the master equation of the second half of the series. Chapter 4 (DPO) solves it in closed form until the reward model and the RL loop cancel out. Chapter 5 (GRPO) changes how the advantage is estimated until the value network disappears. Understand this one chapter, and the next two become "deleting parts" — readable at sight.

3. The Equations

3.1 The reward model: Bradley–Terry

Stage A trains $r_\phi$ with one short loss:

$$\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}}\Big[\log\sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)\Big]$$

• $r_\phi(x,y)$ = a single scalar score per text — in practice a language model whose head is swapped for a single linear layer (num_labels=1)
• $\sigma$ = the sigmoid, turning the score difference into the probability a human picks $y_w$ (the Bradley–Terry model)
• the wider the gap $r_\phi(x,y_w) - r_\phi(x,y_l)$, the lower the loss

The point people overlook and pay for later: this loss sees only the difference of the scores. Replace $r_\phi$ with $r_\phi + c$ for any constant $c$ — the loss doesn't change at all. Which means the absolute scale of a reward model is meaningless and is not pinned down by training. Two runs can yield mean scores of 3.7 and −12.4 that rank identically. This is why you must always standardize rewards before feeding them into PPO (subtract the mean, divide by the std) — hold onto this; it returns in sections 7 and 9.

3.2 The master equation: the RLHF objective

If you memorize a single equation from this whole series, memorize this one:

$$\max_\theta\ \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}\big[r_\phi(x,y)\big] \;-\; \beta\,\mathbb{D}_{\text{KL}}\big(\pi_\theta(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\big)$$

In plain language: "collect as much reward as you can, but every step you walk away from the starting model costs a fine."

• $\pi_\theta$ = the policy, the model being trained — note that $y$ is sampled from $\pi_\theta$ itself; this is the structural difference from SFT, which learns from static data sitting in a file
• $\pi_{\text{ref}}$ = the reference, the starting model (the post-SFT model from chapter 2), frozen throughout training
• $\beta$ = the price per nat of wandering off — the tightness of the leash
• $\mathbb{D}_{\text{KL}}$ = the distributional distance between the policy and the reference

Why this is the master equation of the second half of the series

Chapter 4 (DPO) will prove this equation has a closed-form solution, then flip it inside out until $r_\phi$ and the RL loop both vanish. Chapter 5 (GRPO) will keep the RL skeleton but change how the advantage is computed until $V_\psi$ vanishes. Neither chapter proposes a new objective — they solve this same equation with different tools.

3.3 The PPO clipped surrogate: Stage B's engine

Raw policy gradient (REINFORCE) can use a batch of rollouts for exactly one update before throwing it away — very expensive, because generation is the bottleneck. PPO wants to squeeze the same rollouts for several epochs, so it needs a correction factor (the importance sampling ratio):

$$\rho_t = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}$$

• $s_t$ = the state at position $t$: the prompt plus every token sampled so far
• $a_t$ = the "action": the next token, already sampled during the rollout
• $\pi_{\theta_{\text{old}}}$ = a snapshot of the policy at rollout time — computed once and frozen

Then $\rho_t$ gets pinched with a clip:

$$\mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t\,\hat A_t,\ \text{clip}(\rho_t,\,1-\epsilon,\,1+\epsilon)\,\hat A_t\big)\Big]$$

• $\hat A_t$ = the advantage: "how much better than expected was this token" (defined in the next subsection)
• $\epsilon$ = the width of the trust region (standard value 0.2)

The heart is that min + clip work together with deliberate pessimism: if $\hat A_t$ is positive (a good token), the payoff from pushing $\rho_t$ is capped at $1+\epsilon$ — pushing beyond that earns nothing, gradient zero. But if $\hat A_t$ is negative (a bad token), the min always picks the worse branch — the penalty has no floor. One-sentence summary: gains capped, losses uncapped. The policy therefore moves in small steps that stay close to where it already is.

Don't get confused: there are two "old models," and they are not the same model

$\pi_{\text{ref}}$ in equation 3.2 is frozen for the entire run and serves as the KL leash. $\pi_{\theta_{\text{old}}}$ in equation 3.3 is a snapshot at the latest rollout, refreshed every round, and serves as the trust region. The number-one bug among people writing their own PPO is storing these two in the same variable.

3.4 GAE: computing the advantage without drowning in noise

The advantage is built from the TD error of the value network $V_\psi$:

$$\delta_t = r_t + \gamma V_\psi(s_{t+1}) - V_\psi(s_t)$$ $$\hat A_t = \sum_{l=0}^{\infty} (\gamma\lambda)^l\,\delta_{t+l}$$

• $V_\psi(s_t)$ = the value network's prediction of "from here to the end, how much more reward will be collected" — this is model number four
• $r_t$ = the per-token reward (in our task: a KL penalty at every position, plus the task score at the final token)
• $\gamma$ = the discount factor (LLM work usually uses 1.0)
• $\lambda$ = the bias–variance dial: $\lambda = 0$ trusts $V_\psi$ wholeheartedly (high bias if $V_\psi$ mispredicts), $\lambda = 1$ doesn't trust it at all and waits for the real outcome to the very end (high variance, carrying the noise of the whole trajectory); the popular value is 0.95

Remember this V_ψ well — it is the one GRPO will kill

$V_\psi$ is a model roughly the size of the policy that must be trained alongside it with a loss of its own. If $V_\psi$ predicts garbage, the advantage is garbage, and the policy learns from a garbage signal — PPO's classic failure point. Chapter 5 will answer the question "what if we replaced $V_\psi$ with the mean of a group of answers sampled from the same prompt?" That is the entire GRPO algorithm — deleting the fourth model with a single average.

3.5 The full PPO loss: three terms, two models

Assemble every piece into the one loss the optimizer actually sees (written as a minimization):

$$\mathcal{L}_{\text{PPO}} = -\mathcal{L}^{\text{CLIP}} \;+\; c_1\,\mathbb{E}_t\Big[\big(V_\psi(s_t) - \hat R_t\big)^2\Big] \;-\; c_2\,\mathbb{E}_t\Big[\mathcal{H}\big[\pi_\theta(\cdot \mid s_t)\big]\Big]$$

• first term = the clipped surrogate from 3.3 (negated because we want to maximize it)
• second term = the value loss teaching $V_\psi$ to predict close to the actual return $\hat R_t$; $c_1$ is usually 0.5
• third term = the entropy bonus $\mathcal{H}$, keeping the distribution from collapsing too fast; $c_2$ is usually 0.01
• as for the KL leash from equation 3.2, standard practice folds it into the per-token reward: $r_t \leftarrow r_t - \beta\,(\log\pi_\theta - \log\pi_{\text{ref}})$, which is the approach we take in section 7

Count all the toys that need tuning: 4 models, plus $\epsilon, \beta, \gamma, \lambda, c_1, c_2$, and two separate learning rates. This is why PPO is famous for "run it twice with different seeds, get two completely different stories" — and it is the reason for chapter 4's entire existence.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →