[LLM 5/10] GRPO: Deleting the Value Network and Letting a Group of Answers Be Its Own Baseline

The last chapter ended on DPO's gap: it can only rank answers someone prepared in a file. And chapter 3 paid PPO's full price: 4 models in VRAM, plus an entire value network to train on the side. This chapter combines the good halves of both — the model samples its own answers to learn from, genuine RL — but deletes the value network wholesale, using a statistical observation so simple it's almost irritating nobody framed it sooner: if you sample several answers to the same problem, the group's mean reward is already the baseline the value network was trying to estimate. And if the problem can be graded by code, we need no human preference data at all — zero pairs, zero baht.

Open in Colab05_grpo.ipynb

Thai LLM seriesPart 5 of 10

1. 1Continue Pretraining
2. 2SFT and LoRA
3. 3RLHF and PPO
4. 4DPO: Direct Preference Optimization
5. 5GRPO(you are here)
6. 6Context Distillation
7. 7Model Distillation
8. 8Guardrails
9. 9Benchmarking
10. 10Deployment

1. The Problem

Set the problem up like this: teach Qwen3-0.6B to solve Thai math word problems, where the final answer is a single number that a one-line == can grade.

Walk through the tools of the past three chapters one at a time, and none of them fits this job:

Method	Can the model sample its own answers and learn from them?	Human labels needed	Models in VRAM
SFT (ch. 2)	No — imitates the answer key only	A human-written answer per example	1
PPO (ch. 3)	Yes	Preference pairs to train a reward model	4
DPO (ch. 4)	No — purely offline	Preference pairs	2 (1 with LoRA)

• SFT teaches imitation of a worked solution, but never lets the model try and fail on its own — it never sees which of its own lines of reasoning lead to the right answer.
• PPO lets the model try, but charges you a reward model trained from preference pairs plus an entire value network — for a task whose reward can be written directly as a Python function.
• DPO deletes RL elegantly, but can only rank answers that already exist in the dataset. A math problem needs the model to explore many paths and reinforce the ones that reach the right answer.

So this chapter's question is narrow and sharp: which parts of PPO are truly necessary, and which can be deleted, when our reward can be checked by code?

2. What We're Going to Do

Go back to the value network's job in chapter 3: it exists to answer one question — "on average, roughly what reward should this prompt earn?" — to serve as a baseline subtracted from the actual reward. Answers "better than average" get positive gradient; "worse than average," negative. Without that baseline, the policy gradient is so noisy it can barely train at all.

PPO answers that question by training an entire extra model to predict this average. GRPO answers it by sampling until you can see it with your own eyes:

The core idea of this chapter

Sample $G$ answers from the same prompt and average the group's rewards — that mean is already, by definition, an unbiased estimate of "the expected reward for this prompt." It is the very thing the value network tries to approximate, except it needs no training, no loading, and can never drift wrong. The entire value network can therefore be deleted. And when the reward is checked by code (numeric answer right or wrong), the reward model and the human preference data vanish along with it — zero labels remain.

This is GRPO (Group Relative Policy Optimization), proposed by Shao et al. (2024) in DeepSeekMath, and it is the same engine that trained DeepSeek-R1. The broader approach carries the collective name RLVR (RL with Verifiable Rewards) — RL whose reward comes from a verifier, not from human taste.

And let's set expectations straight from the top of the chapter: current evidence points to this kind of RL mostly "sharpening" ability the base model already has at pass@8 so it surfaces at pass@1, rather than creating new capability from zero. We return to this with measuring instruments in section 9.

3. The Equations

3.1 The group-relative advantage — the whole heart in one line

$$\hat A_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}$$

• $G$ = the number of answers sampled from the same prompt (8 in this chapter)
• $r_i$ = the reward of answer $i$
• Every token of answer $i$ shares the same single $\hat A_i$ across the whole sequence — unlike PPO, which chases per-token advantage through the value network and GAE

In plain language: "is this answer better or worse than my own other attempts at the same problem?" No comparison across problems, no predicting the future — only a competition inside the group.

This short equation has one consequence that matters enormously: if the whole group earns identical rewards (all right or all wrong), every $\hat A_i$ is zero and that batch teaches nothing at all. Remember that sentence — it will become both the chapter's number-one trap and its most important metric.

3.2 The full GRPO objective

$$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\Big\{\min\!\big(\rho_{i,t}\,\hat A_i,\ \operatorname{clip}(\rho_{i,t},\,1-\epsilon,\,1+\epsilon)\,\hat A_i\big) - \beta\,\mathbb{D}_{\text{KL}}\big[\pi_\theta \,\|\, \pi_{\text{ref}}\big]\Big\}\right]$$

where $\rho_{i,t} = \dfrac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}$ is the token's probability ratio against the policy at sampling time.

Read it piece by piece, because every piece has already passed before your eyes in this series:

• $\min(\cdot,\ \operatorname{clip}(\cdot))$ = the same PPO clip from chapter 3, nothing new — keeps steps from wandering too far from the point where the rollouts were sampled
• $\frac{1}{|o_i|}$ = per-token averaging, keeping long answers from outsized influence (think of the length bias from chapter 4)
• $\beta\,\mathbb{D}_{\text{KL}}$ = the same leash as ever, tied to the same $\pi_{\text{ref}}$ as chapters 3 and 4

What deserves the closest reading is what is not in the equation: no $V(s)$, no GAE, no critic loss. The whole line runs on just two models ($\pi_\theta$ and $\pi_{\text{ref}}$) and reward numbers from a verifier.

3.3 The KL term isn't computed directly — meet the k3 estimator

True KL divergence requires summing over the entire vocabulary at every position, which is expensive and unnecessary. GRPO estimates it from the tokens already sampled, using an estimator nicknamed k3:

$$\hat{\mathbb{D}}_{k3} = \frac{\pi_{\text{ref}}(o_{i,t})}{\pi_\theta(o_{i,t})} - \log\frac{\pi_{\text{ref}}(o_{i,t})}{\pi_\theta(o_{i,t})} - 1$$

The question students always ask (and should): why not just use $\log(\pi_\theta/\pi_{\text{ref}})$ directly, when its expectation is already the KL?

The answer: the naive estimator (called k1) is indeed unbiased, but individual samples can go negative — about 40% of samples come out negative, even though KL cannot be negative by definition — and the variance is very high. At realistic batch sizes the estimate swings so hard that the penalty alternately pushes and pulls.

k3 fixes both at once. Let $x = \pi_{\text{ref}}/\pi_\theta$ and observe two facts:

1. The inequality $x - 1 \geq \log x$ always holds, so k3 $= (x-1) - \log x \geq 0$ on every sample
2. $\mathbb{E}_{\pi_\theta}[x] = \sum \pi_\theta \cdot \frac{\pi_{\text{ref}}}{\pi_\theta} = 1$, so the $(x-1)$ term has expectation zero — it is a control variate that cancels the noise of $-\log x$ without touching the expectation

The result is an estimator exactly as unbiased, with variance lower by an order, that can never go negative. Figure 5.3 will show you the difference with your own eyes.

3.4 An advanced note: dividing by std isn't as pure as it looks (Dr.GRPO)

The division by $\operatorname{std}(r_1..r_G)$ in equation 3.1 smuggles in one bias: groups whose rewards are nearly identical (small std — say 7 right out of 8) get their advantages amplified by an enormous factor, while groups that genuinely disagree (large std — which carry the most information) get comparatively muted. The net effect is a gradient tilted toward problems the model nearly already agrees with itself about. The Dr.GRPO work (Liu et al., 2025) proposes dropping the std division entirely, keeping only the mean subtraction — which remains a perfectly correct baseline. The widget in section 4 has a toggle so you can try both.

3.5 Unbiased pass@k — the tool section 9 will need

Sample $n$ answers per problem, get $c$ correct, and ask "given a budget of $k$ tries, would at least one be right?":

$$\widehat{\text{pass@}k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

The fraction at the back is the probability that drawing $k$ from $n$ finds nothing but wrong answers. The formula people commonly misuse is $1-(1-c/n)^k$, which is systematically biased in your favor when $n$ is small (this is why the HumanEval paper by Chen et al. 2021 needed a separate appendix on exactly this). Keep this formula close — it is the yardstick for judging whether GRPO "creates" new capability or merely "sharpens" what exists.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →