[LLM 5/10] GRPO: Deleting the Value Network and Letting a Group of Answers Be Its Own Baseline

The last chapter ended on DPO's gap: it can only rank answers someone prepared in a file. And chapter 3 paid PPO's full price: 4 models in VRAM, plus an entire value network to train on the side. This chapter combines the good halves of both — the model samples its own answers to learn from, genuine RL — but deletes the value network wholesale, using a statistical observation so simple it's almost irritating nobody framed it sooner: if you sample several answers to the same problem, the group's mean reward is already the baseline the value network was trying to estimate. And if the problem can be graded by code, we need no human preference data at all — zero pairs, zero baht.

1. The Problem​

Set the problem up like this: teach Qwen3-0.6B to solve Thai math word problems, where the final answer is a single number that a one-line == can grade.

Walk through the tools of the past three chapters one at a time, and none of them fits this job:

• SFT teaches imitation of a worked solution, but never lets the model try and fail on its own — it never sees which of its own lines of reasoning lead to the right answer.
• PPO lets the model try, but charges you a reward model trained from preference pairs plus an entire value network — for a task whose reward can be written directly as a Python function.
• DPO deletes RL elegantly, but can only rank answers that already exist in the dataset. A math problem needs the model to explore many paths and reinforce the ones that reach the right answer.

So this chapter's question is narrow and sharp: which parts of PPO are truly necessary, and which can be deleted, when our reward can be checked by code?

2. What We're Going to Do​

Go back to the value network's job in chapter 3: it exists to answer one question — "on average, roughly what reward should this prompt earn?" — to serve as a baseline subtracted from the actual reward. Answers "better than average" get positive gradient; "worse than average," negative. Without that baseline, the policy gradient is so noisy it can barely train at all.

PPO answers that question by training an entire extra model to predict this average. GRPO answers it by sampling until you can see it with your own eyes:

This is GRPO (Group Relative Policy Optimization), proposed by Shao et al. (2024) in DeepSeekMath, and it is the same engine that trained DeepSeek-R1. The broader approach carries the collective name RLVR (RL with Verifiable Rewards) — RL whose reward comes from a verifier, not from human taste.

And let's set expectations straight from the top of the chapter: current evidence points to this kind of RL mostly "sharpening" ability the base model already has at pass@8 so it surfaces at pass@1, rather than creating new capability from zero. We return to this with measuring instruments in section 9.

3. The Equations​

3.1 The group-relative advantage — the whole heart in one line​

• $G$ = the number of answers sampled from the same prompt (8 in this chapter)
• $r_i$ = the reward of answer $i$
• Every token of answer $i$ shares the same single $\hat A_i$ across the whole sequence — unlike PPO, which chases per-token advantage through the value network and GAE

In plain language: "is this answer better or worse than my own other attempts at the same problem?" No comparison across problems, no predicting the future — only a competition inside the group.

This short equation has one consequence that matters enormously: if the whole group earns identical rewards (all right or all wrong), every $\hat A_i$ is zero and that batch teaches nothing at all. Remember that sentence — it will become both the chapter's number-one trap and its most important metric.

3.2 The full GRPO objective​

where $\rho_{i,t} = \dfrac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}$ is the token's probability ratio against the policy at sampling time.

Read it piece by piece, because every piece has already passed before your eyes in this series:

• $\min(\cdot,\ \operatorname{clip}(\cdot))$ = the same PPO clip from chapter 3, nothing new — keeps steps from wandering too far from the point where the rollouts were sampled
• $\frac{1}{|o_i|}$ = per-token averaging, keeping long answers from outsized influence (think of the length bias from chapter 4)
• $\beta\,\mathbb{D}_{\text{KL}}$ = the same leash as ever, tied to the same $\pi_{\text{ref}}$ as chapters 3 and 4

What deserves the closest reading is what is not in the equation: no $V(s)$, no GAE, no critic loss. The whole line runs on just two models ($\pi_\theta$ and $\pi_{\text{ref}}$) and reward numbers from a verifier.

3.3 The KL term isn't computed directly — meet the k3 estimator​

True KL divergence requires summing over the entire vocabulary at every position, which is expensive and unnecessary. GRPO estimates it from the tokens already sampled, using an estimator nicknamed k3:

The question students always ask (and should): why not just use $\log(\pi_\theta/\pi_{\text{ref}})$ directly, when its expectation is already the KL?

The answer: the naive estimator (called k1) is indeed unbiased, but individual samples can go negative — about 40% of samples come out negative, even though KL cannot be negative by definition — and the variance is very high. At realistic batch sizes the estimate swings so hard that the penalty alternately pushes and pulls.

k3 fixes both at once. Let $x = \pi_{\text{ref}}/\pi_\theta$ and observe two facts:

1. The inequality $x - 1 \geq \log x$ always holds, so k3 $= (x-1) - \log x \geq 0$ on every sample
2. $\mathbb{E}_{\pi_\theta}[x] = \sum \pi_\theta \cdot \frac{\pi_{\text{ref}}}{\pi_\theta} = 1$, so the $(x-1)$ term has expectation zero — it is a control variate that cancels the noise of $-\log x$ without touching the expectation

The result is an estimator exactly as unbiased, with variance lower by an order, that can never go negative. Figure 5.3 will show you the difference with your own eyes.

3.4 An advanced note: dividing by std isn't as pure as it looks (Dr.GRPO)​

The division by $\operatorname{std}(r_1..r_G)$ in equation 3.1 smuggles in one bias: groups whose rewards are nearly identical (small std — say 7 right out of 8) get their advantages amplified by an enormous factor, while groups that genuinely disagree (large std — which carry the most information) get comparatively muted. The net effect is a gradient tilted toward problems the model nearly already agrees with itself about. The Dr.GRPO work (Liu et al., 2025) proposes dropping the std division entirely, keeping only the mean subtraction — which remains a perfectly correct baseline. The widget in section 4 has a toggle so you can try both.

3.5 Unbiased pass@k — the tool section 9 will need​

Sample $n$ answers per problem, get $c$ correct, and ask "given a budget of $k$ tries, would at least one be right?":

The fraction at the back is the probability that drawing $k$ from $n$ finds nothing but wrong answers. The formula people commonly misuse is $1-(1-c/n)^k$, which is systematically biased in your favor when $n$ is small (this is why the HumanEval paper by Chen et al. 2021 needed a separate appendix on exactly this). Keep this formula close — it is the yardstick for judging whether GRPO "creates" new capability or merely "sharpens" what exists.

4. Seeing the Equations​

What one group teaches — and which groups teach nothing at all​

The left panel is equation 3.1 at work: the two answers that did everything right (r = 1.5) get a strong positive push, while the answer that earned only the format bonus (r = 0.3) gets pushed down despite its positive reward — because the criterion is not "is it good" but "is it better than its groupmates." The right panel is the silent death mode of all of GRPO: identical rewards = zero std = zero learning.

Enter your own rewards and watch the advantages change live — and don't miss unticking the Divide by std box to see the Dr.GRPO difference from section 3.4 with your own eyes:

What GRPO deletes from PPO​

Notice that the two blocks that vanished are exactly the two that had to be trained (the value net) or pre-trained (the reward model). What's left is the policy and the reference — and chapter 4 already taught us LoRA lets those two share base weights. The net model cost of GRPO in this chapter's notebook therefore equals plain SFT.

k3 versus k1: equally unbiased, not equally usable​

The right panel is the practical point: both lines converge to the same answer (both unbiased), but at sample counts matching a real batch (tens to hundreds) the k1 line still swings hard, while k3 is steady enough to serve as a trustworthy penalty from the earliest steps.

5. Setting Up the Environment​

Open Colab and pick Runtime → Change runtime type → T4 GPU (the free tier is enough).

torch_dtype=torch.float16      # not bfloat16
attn_implementation="sdpa"     # not flash_attention_2
fp16=True                      # in GRPOConfig (not bf16=True)

cap = torch.cuda.get_device_capability(0)
print("compute capability:", cap)                    # T4 = (7, 5)
print("native bf16:", cap[0] >= 8)                   # T4 -> False
print("torch says   :", torch.cuda.is_bf16_supported())  # T4 -> True (counts emulation!)

Two models for the price of one — the same recipe as chapter 4​

The policy is the LoRA adapter from chapter 2; $\pi_{\text{ref}}$ is the same base with the adapter off. TRL knows this mechanism natively: if model is a PeftModel, it loads no separate reference.

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
).cuda()

policy = PeftModel.from_pretrained(base, "kobkrit/qwen3-0.6b-th-sft-lora", is_trainable=True)

6. Preparing the Data​

We use VISAI-AI/gsm8k-thai — a Thai translation of the GSM8K math word-problem set. We sample 128 problems from the train split and keep a separate held-out set for evaluation, untouched during training.

from datasets import load_dataset

ds = load_dataset("VISAI-AI/gsm8k-thai", split="train")
ds = ds.shuffle(seed=42).select(range(128))

# The notebook's system prompt is the Thai original of this instruction —
# the model is being trained to reason in Thai:
SYSTEM = "Think step by step inside <think>...</think>, then end with the numeric answer on the last line."

def to_prompt(ex):
    return {
        "prompt": [{"role": "system", "content": SYSTEM},
                   {"role": "user", "content": ex["translated_question"]}],
        "answer": extract_final_int(ex["translated_answer"]),   # the GSM8K answer key sits after "####"
    }

train_ds = ds.map(to_prompt, remove_columns=ds.column_names)

Notice there is no chosen/rejected column and no human label of any kind — only problems and numeric answers. What replaces the labels is three reward functions checked entirely by code:

import re

THAI_DIGITS = str.maketrans("\u0e50\u0e51\u0e52\u0e53\u0e54\u0e55\u0e56\u0e57\u0e58\u0e59",
                            "0123456789")            # Thai digit glyphs 0-9

def extract_final_int(text):
    text = text.translate(THAI_DIGITS)               # in case the model answers in Thai numerals
    tail = text.split("</think>")[-1]                # grade only what follows the thinking
    nums = re.findall(r"-?\d[\d,]*", tail)
    return int(nums[-1].replace(",", "")) if nums else None

def reward_correct(completions, answer, **kwargs):   # +1.0 final answer correct
    return [1.0 if extract_final_int(c) == a else 0.0
            for c, a in zip(completions, answer)]

def reward_format(completions, **kwargs):            # +0.3 has a non-empty <think>
    pat = re.compile(r"<think>.+?</think>", re.DOTALL)
    return [0.3 if pat.search(c) else 0.0 for c in completions]

def reward_thai(completions, **kwargs):              # +0.2 genuinely thinks in Thai
    def th_ratio(s):
        letters = [ch for ch in s if ch.isalpha()]
        return sum("\u0e01" <= ch <= "\u0e5b" for ch in letters) / max(len(letters), 1)   # Thai Unicode block
    return [0.2 if th_ratio(c) > 0.5 else 0.0 for c in completions]

The total reward of one answer is the sum of all three: at most 1.5, at least 0.0. (The THAI_DIGITS table exists because Thai has its own digit glyphs, U+0E50 through U+0E59 — a model that writes the correct answer in Thai numerals still deserves its +1.0. And the U+0E01 through U+0E5B comparison is simply the Thai Unicode block, the same range behind the series' th_ratio metric.)

7. The Main Code​

7.1 GRPOTrainer — this time the trainer is a first-class citizen​

In chapter 3 we had to squint at TRL's PPOTrainer, stuck in semi-experimental status with an API shifting nearly every minor version. GRPOTrainer is a different story entirely: it is the trainer TRL showcases in the wake of R1, and it takes rewards as plain Python functions, no model-wrapping of any kind.

from trl import GRPOConfig, GRPOTrainer

FAST_MODE = True   # default: finishes in ~10 minutes on a free T4
                   # False = the full config used for reported results (~18 minutes)

cfg = GRPOConfig(
    output_dir="grpo-out",
    num_generations=4 if FAST_MODE else 8,          # G — the group size
    max_completion_length=192 if FAST_MODE else 256,
    max_prompt_length=256,
    temperature=1.0,                # do not lower — diversity within the group is the fuel
    beta=0.04,                      # KL weight (computed with k3 from section 3.3)
    epsilon=0.2,                    # the same clip range as PPO
    learning_rate=1e-6,             # lower than DPO still — see the warning below
    per_device_train_batch_size=16, # must divide evenly by num_generations
    gradient_accumulation_steps=2 if FAST_MODE else 4,
    num_train_epochs=1,
    fp16=True,                      # the T4 has no bf16
    logging_steps=1,
)

trainer = GRPOTrainer(
    model=policy,                                        # PeftModel → free reference
    reward_funcs=[reward_correct, reward_format, reward_thai],
    args=cfg,
    train_dataset=train_ds,
)
trainer.train()

Let's do the budget arithmetic in the open: the full config is 128 problems × 8 answers = 1,024 completions, divided by an effective batch of 64 completions per step = 16 optimizer steps — that is genuinely all. Nearly all the remaining time is spent generating roughly 262k tokens (at most) before each step.

7.2 Computing the advantage by hand — all the arithmetic in one place​

So that GRPOTrainer isn't a black box, the notebook has a cell that does equation 3.1's arithmetic in the open:

import torch

def group_advantages(rewards, G, eps=1e-4):
    """rewards: [B], laid out in groups of G from the same prompt"""
    r = rewards.view(-1, G)                    # [B/G, G]
    mean = r.mean(dim=1, keepdim=True)
    std = r.std(dim=1, keepdim=True)
    return ((r - mean) / (std + eps)).view(-1) # Dr.GRPO: delete "/ (std + eps)"

r = torch.tensor([1.5, 0.3, 0.5, 1.5, 0.3, 0.0, 0.3, 0.5])   # the group from figure 5.1
print(group_advantages(r, G=8))
# → [+1.56, -0.55, -0.20, +1.56, -0.55, -1.08, -0.55, -0.20]

These eight lines are everything GRPO adds on top of the plain PPO clip. Compare that to chapter 3's value network + GAE, which had to be trained alongside the policy the whole way — this is the best trade in the series.

8. Results​

The notebook measures 4 things and writes them to results.json:

1. Mean reward per step — should climb (this is what the optimizer sees)
2. The fraction of groups with nonzero std, per step — the metric almost nobody plots
3. pass@1 and pass@8 on held-out, using the unbiased formula from section 3.5, with a Wilson 95% CI
4. Mean completion length per step — read side by side with accuracy

The promised mini-R1 moment​

The DeepSeek-R1 paper has a very famous plot: answer length growing on its own, together with accuracy, with nobody telling the model to think longer — the model discovered by itself that writing out its reasoning in more detail brings reward. Our notebook plots the same pair (completion length and accuracy per step) at miniature scale. If you see both lines drift upward together even slightly, that is the same mechanism as R1 in your own test tube. And if length grows while accuracy stays flat, suspect reward hacking first, always (trap 1, section 9).

9. Comparison​

Four chapters, four methods, measured on the same task (the same held-out Thai math problems) — this table is where the whole series pays out, because the rightmost column has never appeared in any chapter before:

Read this table right to left: the series' arc is a progressive shedding of human labels — from an answer for every example → preference pairs → zero — with the machinery underneath growing simpler as well. The one condition that makes the last column zero is that the task must be checkable by code — engrave that somewhere.

Does RL create new capability, or sharpen the old?​

The logic behind this figure is stronger than it looks: GRPO learns from advantages, which can only be nonzero when at least one answer in the group does better than its peers — meaning a problem the base model never solves no matter how it samples (p = 0) can never send a learning signal into the system. What RLVR does well is take ability scattered across pass@8 and concentrate it at pass@1. Research in 2025 (Yue et al.) even measured that at very large $k$, the base model can beat the post-RL model. This does not make GRPO worthless — real users get one answer; pass@1 is the number that matters — but it does mean you shouldn't read a climbing reward curve and conclude the model "got smarter." Mostly, it got steadier.

Traps to watch for​

1. Reward hacking: farming +0.3 with an empty <think> The notebook's first version of the format reward used the regex <think>.*?</think> (the crucial detail: .*? accepts the empty string). Within a few steps the model discovered that printing a bare, empty <think></think> and then guessing a number collects +0.3 for free every time — far cheaper than actually thinking. Mean reward climbed beautifully; accuracy didn't move. The notebook keeps the caught-in-the-act samples for you to see, then fixes the pattern to .+?, forcing real content. The lesson: the model does not optimize what you meant. It optimizes what you wrote.

2. Identical rewards across the whole group → figure 5.1's right panel, the lesson The smaller the group, the likelier every answer earns the same reward — at $G = 2$, two coins land the same way very often. Groups smaller than 4 burn a large share of compute for nothing, and a mean estimated from two samples is high-noise anyway. $G = 8$ is the widely used balance point (our FAST mode accepts dropping to 4 in exchange for time — and says so out loud).

3. Temperature too low = killing diversity at the source Lower the temperature and all 8 answers come out nearly identical → identical rewards → back to trap 2. Never carry inference habits (low temperature for stability) into rollout collection. We set temperature=1.0 because diversity within the group is the fuel of the entire learning system.

4. The bottleneck is generation, not backprop — budget the right pile If a run is slow, don't reach for batch size or the optimizer first — look at the ~262k-token budget in section 5. The levers that work, strongest first: lower max_completion_length, lower num_generations, use fewer problems. (Production systems solve this with an inference engine like vLLM, which TRL can attach to, but that is beyond free Colab.)

10. Summary​

• The group mean is a baseline that needs no training — sample $G$ answers from the same prompt, and PPO's entire value network becomes unnecessary
• Code-checkable rewards = zero human labels — the point the series has been climbing toward since chapter 2
• Advantage is relative within the group — an answer with positive reward can still be pushed down, if its groupmates did better
• A group with identical rewards teaches nothing — always plot the fraction of groups with std > 0; it is the line between "saturated" and "stopped learning long ago with nobody noticing"
• k3 makes the KL penalty usable at small batch sizes — as unbiased as the log-ratio, but never negative and far lower variance
• Dividing by std hides a bias — Dr.GRPO cuts it, keeping only the mean subtraction; try it yourself in the section 4 widget
• Reward shaping is necessary and dangerous — sub-rewards prevent all-zero groups early on, but open the door to score farming
• RLVR mostly "sharpens," it doesn't "create" — pass@1 climbs toward the old pass@8 ceiling; measure both, always

Next chapter: Context Distillation — that long system prompt you pay for on every single call to the model: how to distill it into the weights so the model behaves accordingly without ever seeing the prompt again.

References​

1. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — DeepSeekMath: where GRPO comes from
2. DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — R1: RL on verifiable rewards at real scale
3. Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective — Dr.GRPO: the std-division bias discussed in section 3
4. Ahmadian et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs — plain REINFORCE may suffice -- read with the value-network deletion
5. Schulman et al. (2017). Proximal Policy Optimization Algorithms — the original PPO paper: the clipped surrogate in section 3
6. Chen et al. (2021). Evaluating Large Language Models Trained on Code — the unbiased pass@k estimator used in section 9

The writing, code and notebooks in this series are licensed under CC BY-NC-SA 4.0 — reuse and adapt them freely with attribution, for non-commercial purposes, and share your adaptations under the same terms. Third-party models and datasets referenced here keep their own licences.