3 posts tagged with "alignment"

In chapter 2 we taught the model by "imitating the answer key" one token at a time. But the qualities that make an AI assistant genuinely usable — correct, polite, not making things up, not sliding into English — have no answer key to imitate, and cannot be written directly as a loss function. This chapter is the most classical answer to that problem: RLHF (Reinforcement Learning from Human Feedback) with PPO. We will train a real reward model from Thai preference pairs, then write the PPO loop ourselves from scratch in about 120 lines, and close with my favorite experiment in the series: taking off the KL leash and watching the model cheat the reward live. This is deliberately the heaviest chapter of the series, because chapter 4 (DPO) and chapter 5 (GRPO) both start from this chapter's equations and each choose a different piece to delete.

In the last chapter we did RLHF with PPO, and you saw how many moving parts it has — a separate reward model to train, four models resident in VRAM at once, a dozen-plus PPO hyperparameters to tune, and if the reward model is off, the policy finds a shortcut that games the score. In this chapter we'll accomplish the same thing with an ordinary supervised training loop — no reward model, no RL. And crucially, this is not an approximation. We'll prove algebraically that those two pieces genuinely cancel out.

The last chapter ended on DPO's gap: it can only rank answers someone prepared in a file. And chapter 3 paid PPO's full price: 4 models in VRAM, plus an entire value network to train on the side. This chapter combines the good halves of both — the model samples its own answers to learn from, genuine RL — but deletes the value network wholesale, using a statistical observation so simple it's almost irritating nobody framed it sooner: if you sample several answers to the same problem, the group's mean reward is already the baseline the value network was trying to estimate. And if the problem can be graded by code, we need no human preference data at all — zero pairs, zero baht.