[LLM 4/10] DPO: When a Language Model Becomes Its Own Reward Model

In the last chapter we did RLHF with PPO, and you saw how many moving parts it has — a separate reward model to train, four models resident in VRAM at once, a dozen-plus PPO hyperparameters to tune, and if the reward model is off, the policy finds a shortcut that games the score. In this chapter we'll accomplish the same thing with an ordinary supervised training loop — no reward model, no RL. And crucially, this is not an approximation. We'll prove algebraically that those two pieces genuinely cancel out.

Open in Colab04_dpo.ipynb

Thai LLM seriesPart 4 of 10

1. 1Continue Pretraining
2. 2SFT and LoRA
3. 3RLHF and PPO
4. 4DPO: Direct Preference Optimization(you are here)
5. 5GRPO
6. 6Context Distillation
7. 7Model Distillation
8. 8Guardrails
9. 9Benchmarking
10. 10Deployment

1. The Problem

Say you want your AI assistant to "always answer in Thai" — sounds simple. Now try writing that as a loss function. You can't.

This is the central problem of alignment: answer quality cannot be written as an equation. "More polite," "more natural," "doesn't drift into English" — there is no single correct answer key. There is only comparison: show a human two answers and ask which one they prefer. So the data comes in threes: a prompt $x$, a preferred answer $y_w$ (chosen), and a dispreferred one $y_l$ (rejected).

PPO-style RLHF solves this by taking a two-step detour:

1. Train a reward model $r_\phi(x,y)$ to imitate human preferences
2. Use RL to push the policy toward high scores from that reward model

The detour has a price:

Problem with RLHF/PPO	What it costs you in practice
An extra model to train	More steps, more ways to fail, more time
Four models loaded at once	policy + ref + reward + value — VRAM explodes
Reward hacking	The model finds a loophole that scores well without humans liking it any more
PPO is hyperparameter-sensitive	Two runs with different seeds can tell completely different stories

So this chapter asks one short question: can we skip steps 1 and 2 entirely?

2. What We're Going to Do

Yes, we can — and the reason is beautiful.

The starting point is the observation that the KL-constrained RLHF objective has a closed-form solution. We already know exactly what the optimal policy looks like, without running a single step of RL. Knowing that, we invert the equation — instead of asking "what policy does this reward produce?" we ask "what reward does this policy imply?"

The core idea of this chapter

Once you invert the equation, the language model already is a reward model, implicitly. The reward model and the RL loop aren't "approximated away" — they cancel out algebraically. What's left is an ordinary supervised loss function you can train with a single Trainer.

This is DPO (Direct Preference Optimization), proposed by Rafailov et al. (2023). The "Direct" comes from optimizing on preference data directly, with no intermediary.

3. The Equations

3.1 Setting up: the RLHF objective

$$\max_{\pi}\ \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi(\cdot|x)}\big[r(x,y)\big] - \beta\,\mathbb{D}_{\text{KL}}\big[\pi(y|x)\,\|\,\pi_{\text{ref}}(y|x)\big]$$

In plain language: "maximize the reward, but don't wander too far from where you started."

• $\pi$ = the policy, i.e. the model we're training
• $\pi_{\text{ref}}$ = the reference policy, i.e. the starting model (here, the post-SFT model from chapter 2)
• $r(x,y)$ = the reward for answer $y$ given prompt $x$
• $\beta$ = how tight the leash is; higher values pull harder back toward $\pi_{\text{ref}}$

The KL term is not decoration. Without it the model runs off to wherever reward is highest and the language falls apart.

3.2 Step 1 — the closed-form solution

That problem can be solved by hand (it amounts to finding the distribution that minimizes KL against a target distribution), giving

$$\pi^*(y|x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right)$$

• $Z(x) = \sum_{y}\pi_{\text{ref}}(y|x)\exp\!\big(r(x,y)/\beta\big)$ is the partition function, the denominator that makes everything sum to 1
• Note that $Z(x)$ depends only on $x$, not on $y$ — hold on to that sentence, it's about to become the hero of the story

The intuition: the optimal policy is the original model, reweighted by $\exp(r/\beta)$. High-reward answers get their probability amplified, low-reward answers get pushed down, but everything starts from the original shape of $\pi_{\text{ref}}$.

In practice we can't compute $Z(x)$, because it requires summing over every possible answer in the universe. That's why people reach for RL — and it's exactly why DPO doesn't have to.

3.3 Step 2 — invert the equation to get the reward

Take the log of both sides and rearrange:

$$r(x,y) = \beta\log\frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta\log Z(x)$$

This line is the crux: any reward function can be rewritten in terms of the optimal policy and the starting policy. Which means that if we have two models, we can compute their implicit reward immediately, without ever training a reward model.

3.4 Step 3 — substitute into Bradley-Terry and $Z(x)$ vanishes

The standard model of preference is Bradley-Terry: the probability a human picks $y_w$ over $y_l$ is

$$p(y_w \succ y_l \mid x) = \sigma\big(r(x,y_w) - r(x,y_l)\big)$$

where $\sigma$ is the sigmoid. Notice that reward appears in this equation only as a difference. Substitute equation 3.3 — $\beta\log Z(x)$ appears identically on both sides because it's the same $x$ — so it cancels out.

$$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x,y_w,y_l)}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$$

This is the sentence the whole article exists to say

The uncomputable thing ($Z(x)$) disappears, because Bradley-Terry only cares about the difference of rewards. What remains is the log-probability of two models on text we already have, obtainable with an ordinary forward pass. No sampling of answers, no rollouts, no value function — DPO is supervised learning, all the way down.

3.5 The gradient — where the intuition lives

$$\nabla_\theta\mathcal{L}_{\text{DPO}} = -\beta\,\mathbb{E}\left[\sigma(\hat r_l - \hat r_w)\left(\nabla_\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x)\right)\right]$$

where $\hat r = \beta\log\big(\pi_\theta/\pi_{\text{ref}}\big)$ is called the implicit reward.

Read it piece by piece:

• The right-hand parenthesis = the direction: push the log-prob of $y_w$ up and the log-prob of $y_l$ down, simultaneously
• $\sigma(\hat r_l - \hat r_w)$ = the weight: "how badly is the model ranking this pair?"

That weight is the most important teaching point here. If the model already ranks the pair correctly ($\hat r_w$ clearly greater than $\hat r_l$), then $\sigma(\hat r_l - \hat r_w)$ approaches zero and the pair contributes almost no gradient. DPO therefore focuses on its own mistakes automatically, with nobody curating the data for it.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →