[LLM 4/10] DPO: When a Language Model Becomes Its Own Reward Model

In the last chapter we did RLHF with PPO, and you saw how many moving parts it has — a separate reward model to train, four models resident in VRAM at once, a dozen-plus PPO hyperparameters to tune, and if the reward model is off, the policy finds a shortcut that games the score. In this chapter we'll accomplish the same thing with an ordinary supervised training loop — no reward model, no RL. And crucially, this is not an approximation. We'll prove algebraically that those two pieces genuinely cancel out.

1. The Problem​

Say you want your AI assistant to "always answer in Thai" — sounds simple. Now try writing that as a loss function. You can't.

This is the central problem of alignment: answer quality cannot be written as an equation. "More polite," "more natural," "doesn't drift into English" — there is no single correct answer key. There is only comparison: show a human two answers and ask which one they prefer. So the data comes in threes: a prompt $x$, a preferred answer $y_w$ (chosen), and a dispreferred one $y_l$ (rejected).

PPO-style RLHF solves this by taking a two-step detour:

1. Train a reward model $r_\phi(x,y)$ to imitate human preferences
2. Use RL to push the policy toward high scores from that reward model

The detour has a price:

So this chapter asks one short question: can we skip steps 1 and 2 entirely?

2. What We're Going to Do​

Yes, we can — and the reason is beautiful.

The starting point is the observation that the KL-constrained RLHF objective has a closed-form solution. We already know exactly what the optimal policy looks like, without running a single step of RL. Knowing that, we invert the equation — instead of asking "what policy does this reward produce?" we ask "what reward does this policy imply?"

This is DPO (Direct Preference Optimization), proposed by Rafailov et al. (2023). The "Direct" comes from optimizing on preference data directly, with no intermediary.

3. The Equations​

3.1 Setting up: the RLHF objective​

In plain language: "maximize the reward, but don't wander too far from where you started."

• $\pi$ = the policy, i.e. the model we're training
• $\pi_{\text{ref}}$ = the reference policy, i.e. the starting model (here, the post-SFT model from chapter 2)
• $r(x,y)$ = the reward for answer $y$ given prompt $x$
• $\beta$ = how tight the leash is; higher values pull harder back toward $\pi_{\text{ref}}$

The KL term is not decoration. Without it the model runs off to wherever reward is highest and the language falls apart.

3.2 Step 1 — the closed-form solution​

That problem can be solved by hand (it amounts to finding the distribution that minimizes KL against a target distribution), giving

• $Z(x) = \sum_{y}\pi_{\text{ref}}(y|x)\exp\!\big(r(x,y)/\beta\big)$ is the partition function, the denominator that makes everything sum to 1
• Note that $Z(x)$ depends only on $x$, not on $y$ — hold on to that sentence, it's about to become the hero of the story

The intuition: the optimal policy is the original model, reweighted by $\exp(r/\beta)$. High-reward answers get their probability amplified, low-reward answers get pushed down, but everything starts from the original shape of $\pi_{\text{ref}}$.

In practice we can't compute $Z(x)$, because it requires summing over every possible answer in the universe. That's why people reach for RL — and it's exactly why DPO doesn't have to.

3.3 Step 2 — invert the equation to get the reward​

Take the log of both sides and rearrange:

This line is the crux: any reward function can be rewritten in terms of the optimal policy and the starting policy. Which means that if we have two models, we can compute their implicit reward immediately, without ever training a reward model.

3.4 Step 3 — substitute into Bradley-Terry and $Z(x)$ vanishes​

The standard model of preference is Bradley-Terry: the probability a human picks $y_w$ over $y_l$ is

where $\sigma$ is the sigmoid. Notice that reward appears in this equation only as a difference. Substitute equation 3.3 — $\beta\log Z(x)$ appears identically on both sides because it's the same $x$ — so it cancels out.

3.5 The gradient — where the intuition lives​

where $\hat r = \beta\log\big(\pi_\theta/\pi_{\text{ref}}\big)$ is called the implicit reward.

Read it piece by piece:

• The right-hand parenthesis = the direction: push the log-prob of $y_w$ up and the log-prob of $y_l$ down, simultaneously
• $\sigma(\hat r_l - \hat r_w)$ = the weight: "how badly is the model ranking this pair?"

That weight is the most important teaching point here. If the model already ranks the pair correctly ($\hat r_w$ clearly greater than $\hat r_l$), then $\sigma(\hat r_l - \hat r_w)$ approaches zero and the pair contributes almost no gradient. DPO therefore focuses on its own mistakes automatically, with nobody curating the data for it.

4. Seeing the Equations​

Bradley-Terry: from a reward difference to a probability​

Where the difference is zero the probability is exactly 0.5 — "the model has no opinion." And because the curve only cares about the difference, adding a constant to both rewards changes nothing at all. That is the geometric reason $Z(x)$ can cancel.

Loss and the gradient weight​

Let $\Delta = \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$. Then the loss is $-\log\sigma(\beta\Delta)$ and the gradient weight is $\sigma(-\beta\Delta)$.

The right panel is the one to study. Once the margin is strongly positive, the weight converges to zero — that pair has "graduated." And the higher $\beta$ is, the steeper the curve: the model both learns fast and "stops learning" fast. At $\beta = 1.0$, pairs with a margin past 4 have essentially no gradient left. At $\beta = 0.1$ the curve is far flatter, so the model keeps collecting gradient from every pair — slower, but steadier.

$\beta$ controls how far the model can drift from where it started​

Read this one right to left: at $\beta = 0.1$ nearly all the probability mass collapses onto $y_5$, the highest-reward answer. That is mode collapse — a great score with all diversity wiped out. At $\beta = 10$ the bars nearly overlap the dashed line, meaning almost nothing was learned. $\beta$ isn't a hyperparameter you "tune for lowest loss." It is a choice of where to sit on the trade-off between following preferences and keeping your original identity.

Drag the slider yourself and watch how loss and gradient change shape:

5. Setting Up the Environment​

Open Colab and pick Runtime → Change runtime type → T4 GPU (the free tier suffices).

torch_dtype=torch.float16      # not bfloat16
attn_implementation="sdpa"     # not flash_attention_2
fp16=True                      # in DPOConfig (not bf16=True)

cap = torch.cuda.get_device_capability(0)
print("compute capability:", cap)                    # T4 = (7, 5)
print("native bf16:", cap[0] >= 8)                   # T4 -> False
print("torch says   :", torch.cuda.is_bf16_supported())  # T4 -> True (counts emulation!)

A reference model that costs zero extra VRAM​

DPO needs both $\pi_\theta$ and $\pi_{\text{ref}}$, which sounds like loading two models. But we're building on the LoRA adapter from chapter 2, which lets both share the same base weights.

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
).cuda()

policy = PeftModel.from_pretrained(base, "kobkrit/qwen3-0.6b-th-sft-lora", is_trainable=True)

policy is base + adapter, while $\pi_{\text{ref}}$ is that same base with the adapter switched off. Just call it inside the policy.disable_adapter() context manager — nothing extra to load.

6. Preparing the Data​

DPO data needs exactly three columns: prompt, chosen, rejected.

Set 1 — iapp/dpo_thai_tutorial (100 pairs, Apache-2.0) A dataset I built myself for this series and released under Apache-2.0 so anyone can reuse it. Hand-curated Thai preference pairs, emphasizing politeness and natural-sounding language.

Set 2 — roughly 400 pairs built from scratch out of airesearch/wangchanx-seed-free-synthetic-instruct-thai-120k. The construction is about as direct as it gets:

• chosen = the Thai reference answer shipped with the dataset
• rejected = the base model's own output under greedy decoding

def build_rejected(prompt):
    with torch.no_grad(), policy.disable_adapter():
        out = policy.generate(**tok(prompt, return_tensors="pt").to("cuda"),
                              max_new_tokens=192, do_sample=False)
    return tok.decode(out[0], skip_special_tokens=True)[len(prompt):]

That gives roughly 500 pairs, with 15% held out for evaluation and never touched during training.

7. The Main Code​

The point of this section isn't calling a library. It's writing the DPO loss by hand and then proving it matches the real thing. If the equations in section 3 didn't convince you, this code will.

7.1 Twenty-five lines, written from scratch​

import torch, torch.nn.functional as F

def seq_logp(model, input_ids, attention_mask, labels):
    """Sum of log-probs over the ANSWER ONLY (prompt tokens are masked to -100)"""
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits
    logits = logits[:, :-1, :]                   # position t predicts the token at t+1
    target = labels[:, 1:]                       # so the targets shift by 1
    mask = target.ne(-100)                       # count answer tokens only
    target = target.masked_fill(~mask, 0)        # keep gather from breaking on -100
    logp = torch.log_softmax(logits.float(), dim=-1)
    tokp = logp.gather(-1, target.unsqueeze(-1)).squeeze(-1)
    return (tokp * mask).sum(-1)                 # [B] — a sum, not a mean

def dpo_loss(policy, batch, beta=0.1):
    pi_w = seq_logp(policy, batch["chosen_ids"],   batch["chosen_mask"],   batch["chosen_labels"])
    pi_l = seq_logp(policy, batch["rejected_ids"], batch["rejected_mask"], batch["rejected_labels"])

    with torch.no_grad(), policy.disable_adapter():        # reference: adapter off + no gradient
        ref_w = seq_logp(policy, batch["chosen_ids"],   batch["chosen_mask"],   batch["chosen_labels"])
        ref_l = seq_logp(policy, batch["rejected_ids"], batch["rejected_mask"], batch["rejected_labels"])

    delta = (pi_w - ref_w) - (pi_l - ref_l)                # the Δ from section 4
    loss = -F.logsigmoid(beta * delta).mean()              # equation 3.4, literally

    r_w = beta * (pi_w - ref_w).detach()                   # implicit reward of chosen
    r_l = beta * (pi_l - ref_l).detach()                   # implicit reward of rejected
    return loss, r_w, r_l

All of DPO is right there. Nothing else is hiding. The delta line is equation 3.4 transcribed one-to-one, and -F.logsigmoid(beta * delta) is the entire loss.

7.2 Proving it matches TRL​

from trl import DPOTrainer, DPOConfig

cfg = DPOConfig(
    output_dir="dpo-out",
    beta=0.1,
    loss_type="sigmoid",              # must match the formula we wrote by hand
    label_smoothing=0.0,              # anything nonzero and it stops being equation 3.4
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,    # effective batch = 16
    num_train_epochs=2,
    learning_rate=5e-6,               # much lower than SFT — see the warning below
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    max_length=768,
    max_prompt_length=256,
    fp16=True,                        # the T4 has no bf16
    logging_steps=5,
)

trainer = DPOTrainer(model=policy, args=cfg, train_dataset=train_ds, processing_class=tok)

# Critical: a freshly created LoRA has lora_B = 0, so the policy is exactly the
# reference. The margin is zero and BOTH implementations return ln 2 — even if the
# formula is wrong. Perturb the weights first or the assert proves nothing.
with torch.no_grad():
    for name, p in policy.named_parameters():
        if "lora_B" in name:
            p.add_(torch.randn_like(p) * 0.01)

loss_manual, _, _ = dpo_loss(policy, batch, beta=0.1)
loss_trl = trainer.compute_loss(policy, trl_batch)

assert torch.allclose(loss_manual, loss_trl, atol=1e-4)
print("match:", loss_manual.item(), loss_trl.item())

Total training time on a T4 is around 9 minutes for 500 pairs over 2 epochs.

8. Results​

The notebook measures three things and writes them to results.json:

1. Held-out preference accuracy — the fraction of pairs where the implicit reward of chosen exceeds rejected, with a Wilson 95% CI
2. The distribution of implicit reward margins ($\hat r_w - \hat r_l$) — the whole distribution, not just the mean
3. th_ratio — the fraction of Thai characters in the model's generated answers, this series' standing metric for catching silent drift into English

The thing that will alarm you the first time you read the logs​

During training you'll see rewards/chosen and rewards/rejected in TRL's logs, and what happens nearly every time is that both values drift negative together while rewards/margins keeps widening.

9. Comparison​

Next chapter: GRPO — when ranking what already exists stops being enough, and we let the model sample several of its own answers and compare them against each other, with no PPO-style value function.

References​

1. Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model — the original DPO paper -- the derivation in section 3
2. Bradley & Terry (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons — the Bradley-Terry model every reward model rests on
3. Azar et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences — IPO: identifies DPO's overfitting-to-preferences weakness
4. Ethayarajh et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization — KTO: an alternative that needs no chosen/rejected pairs
5. Park et al. (2024). Disentangling Length from Quality in Direct Preference Optimization — the DPO length bias that section 9 warns about
6. Tang et al. (2024). Understanding the performance gap between online and offline alignment algorithms — why offline (DPO) trails online (PPO/GRPO)
7. Ouyang et al. (2022). Training language models to follow instructions with human feedback — InstructGPT: the origin of the whole SFT -> RM -> PPO pipeline

The writing, code and notebooks in this series are licensed under CC BY-NC-SA 4.0 — reuse and adapt them freely with attribution, for non-commercial purposes, and share your adaptations under the same terms. Third-party models and datasets referenced here keep their own licences.