[LLM 9/10] Benchmarking: An Accuracy Number Without a Confidence Interval Is a Rumor

Since chapter 1, this series has repeated the same sentence every time it reports a number: "accuracy without a confidence interval is not an experimental result, it's a rumor" — and promised a full explanation in chapter 9. This is that chapter. We won't train a single step. Instead we'll build a three-mode evaluation system from scratch, put every checkpoint trained across the series on real Thai exams, and prove that the settings nobody writes down in papers move the very same model's score by more than the leaderboard gaps people argue about.

Open in Colab09_benchmarking.ipynb

Thai LLM seriesPart 9 of 10

1. 1Continue Pretraining
2. 2SFT and LoRA
3. 3RLHF and PPO
4. 4DPO: Direct Preference Optimization
5. 5GRPO
6. 6Context Distillation
7. 7Model Distillation
8. 8Guardrails
9. 9Benchmarking(you are here)
10. 10Deployment

1. The Problem

Here's a sentence you can find any week of the year: "Model X scored 71.2% on ThaiExam, beating model Y at 69.8%."

The only question worth asking is out of how many questions. If the test set has 100, the 95% confidence interval on each of those numbers is roughly ±8–10 points. Which means 71.2% and 69.8% are the same number that happened to come out of the randomness differently. Declaring a winner from a 1.4-point gap on a 100-question set is no different from flipping a coin ten times and concluding it's biased.

And the problem doesn't stop at test-set size, because a single "benchmark score" is produced by layers of decisions that almost nobody reports:

Hidden decision	Effect on the score
Scoring by log-likelihood or generatively	can move it several points
Length-normalizing the choices or not ($\gamma$)	can reorder a leaderboard
0-shot or 5-shot, and how the template is written	can move it several points
Which models get the chat template	biases toward one model or another
Whether the exam leaked into the training data (contamination)	inflates the whole bar

One number concealing five layers of decisions, plus an error bar nobody draws — that is the thing we call a leaderboard.

2. What We're Going to Do

This chapter has no training at all, and that is its strength, not its weakness — evaluation is a different kind of work from training, and it deserves a chapter of its own.

We'll do four things:

1. Write a three-mode scoring system from scratch — log-likelihood multiple-choice, generative exact-match (with Thai-specific normalization), and LLM-as-judge with a written rubric — then show that the three modes give the same model different scores
2. Put every checkpoint from chapters 1–8 on the same axis, measured with real Thai exams (scb10x/thai_exam), Thai math word problems (VISAI-AI/gsm8k-thai), and the series' own KobEval-TH
3. Attach a Wilson 95% CI to every number and see which of the series' conclusions survive their error bars
4. Try to reproduce a public leaderboard number for Qwen3-0.6B on ThaiExam — and if ours doesn't match, hunt down the reason instead of staying quiet

The core idea of this chapter

A benchmark score is not a property of the model. It is a property of (model × scoring method × exam set × question count). Reporting the number without reporting the other three is reporting one quarter of the truth, and a confidence interval is the minimum price of admission for the phrase "experimental result."

3. The Equations

3.1 Wilson score interval — the series' standing CI

If you get $s$ right out of $n$ questions, $\hat p = s/n$, the Wilson interval at level $z$ (95% → $z = 1.96$) is

$$\frac{\hat p + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \;\pm\; \frac{z}{1 + \frac{z^2}{n}}\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}$$

Why not the easier-to-remember normal approximation ($\hat p \pm z\sqrt{\hat p(1-\hat p)/n}$)? Because it breaks exactly where we need it most: near the edges 0 and 1, and at small $n$.

Take a real case from chapter 8: the guardrail let 0 dangerous requests through out of a 30-question test set.

from kobeval import wilson_ci      # the same function used since chapter 1

wilson_ci(0, 30)                   # → (0.000, 0.114)

• Normal approximation: $0 \pm 1.96\sqrt{0 \cdot 1/30} = 0 \pm 0$ — an interval of zero width, as if we were 100% certain the leak rate is 0% after watching just 30 examples
• Wilson: $[0\%,\ 11.4\%]$ — "in the 30 tries we've watched, nothing leaked, but the true rate could be as high as 11%"

The second interval is the honest sentence. The first is a lie the formula manufactures automatically. Notice in the formula how Wilson pulls the midpoint toward $1/2$ with the $z^2/2n$ term (like adding $z^2 \approx 4$ phantom questions, half right, half wrong) and how the $z^2/4n^2$ term keeps the width from collapsing to zero — this is why kobeval has used Wilson all series long.

3.2 Length-normalised multiple-choice scoring — the number that quietly decides leaderboards

Log-likelihood mode has the model read the question $q$ and compare the probability of each full choice $c_1,\dots,c_m$:

$$\hat i = \arg\max_i \frac{\log p_\theta(c_i \mid q)}{|c_i|^{\gamma}}$$

• $\gamma = 0$ → raw total log-prob, which is biased toward short choices (fewer tokens = fewer probability multiplications)
• $\gamma = 1$ → divide by token count, i.e. mean log-prob per token

These two lines are acc and acc_norm in lm-evaluation-harness, and on real benchmarks they give different scores and sometimes reorder the models — when two papers report different ThaiExam numbers, one of the top causes is a different $\gamma$, with neither paper ever writing down which value it used.

3.3 Unbiased pass@k — for problems that can be auto-checked

Math and code problems let us sample several answers and ask "was any of them right?" Sample $n$ times, get $c$ correct; the correct estimator of pass@$k$ is

$$\widehat{\text{pass@}k} = 1 - \binom{n-c}{k}\Big/\binom{n}{k}$$

The intuitive estimator $1 - (1 - c/n)^k$ is biased: the function $f(p) = 1-(1-p)^k$ is concave in $p$, so by Jensen's inequality $\mathbb{E}[f(\hat p)] \le f(p)$ — plug an estimate into a nonlinear function and the expectation no longer matches the truth. The binomial formula above reads as "the fraction of $k$-subsets of the $n$ samples that are all wrong," and it can be proven exactly unbiased.

3.4 McNemar's test — comparing two models on the same exam

Models A and B take the same exam. Let $b$ = questions A got right but B got wrong, $c$ = questions B got right but A got wrong:

$$\chi^2 = \frac{(|b-c|-1)^2}{b+c}$$

Compare against $\chi^2_1$ (with continuity correction). The key word is paired: the questions both got right and the questions both got wrong appear nowhere in the formula, because they say nothing about who's better. If you compare two accuracies with a t-test as if they came from different exams, you throw away the "same question" structure and then need many times more questions for the same power.

3.5 Contamination check — did the exam leak into the training data

For an exam question $x$ and a training corpus $\mathcal{D}$, define the $k$-gram overlap rate:

$$\text{overlap}_k(x) = \frac{\left|G_k(x) \cap G_k(\mathcal{D})\right|}{\left|G_k(x)\right|}$$

where $G_k(\cdot)$ is the set of all $k$-grams. For Thai we use character-level k-grams (e.g. 20 characters), because Thai word segmentation is ambiguous. If $\text{overlap}_k$ is high (say above 0.7), suspect the model has already "seen the answer key" — its score on that question measures memory, not ability.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →