[LLM 9/10] Benchmarking: An Accuracy Number Without a Confidence Interval Is a Rumor

Since chapter 1, this series has repeated the same sentence every time it reports a number: "accuracy without a confidence interval is not an experimental result, it's a rumor" — and promised a full explanation in chapter 9. This is that chapter. We won't train a single step. Instead we'll build a three-mode evaluation system from scratch, put every checkpoint trained across the series on real Thai exams, and prove that the settings nobody writes down in papers move the very same model's score by more than the leaderboard gaps people argue about.

1. The Problem​

Here's a sentence you can find any week of the year: "Model X scored 71.2% on ThaiExam, beating model Y at 69.8%."

The only question worth asking is out of how many questions. If the test set has 100, the 95% confidence interval on each of those numbers is roughly ±8–10 points. Which means 71.2% and 69.8% are the same number that happened to come out of the randomness differently. Declaring a winner from a 1.4-point gap on a 100-question set is no different from flipping a coin ten times and concluding it's biased.

And the problem doesn't stop at test-set size, because a single "benchmark score" is produced by layers of decisions that almost nobody reports:

One number concealing five layers of decisions, plus an error bar nobody draws — that is the thing we call a leaderboard.

2. What We're Going to Do​

This chapter has no training at all, and that is its strength, not its weakness — evaluation is a different kind of work from training, and it deserves a chapter of its own.

We'll do four things:

1. Write a three-mode scoring system from scratch — log-likelihood multiple-choice, generative exact-match (with Thai-specific normalization), and LLM-as-judge with a written rubric — then show that the three modes give the same model different scores
2. Put every checkpoint from chapters 1–8 on the same axis, measured with real Thai exams (scb10x/thai_exam), Thai math word problems (VISAI-AI/gsm8k-thai), and the series' own KobEval-TH
3. Attach a Wilson 95% CI to every number and see which of the series' conclusions survive their error bars
4. Try to reproduce a public leaderboard number for Qwen3-0.6B on ThaiExam — and if ours doesn't match, hunt down the reason instead of staying quiet

3. The Equations​

3.1 Wilson score interval — the series' standing CI​

If you get $s$ right out of $n$ questions, $\hat p = s/n$, the Wilson interval at level $z$ (95% → $z = 1.96$) is

Why not the easier-to-remember normal approximation ($\hat p \pm z\sqrt{\hat p(1-\hat p)/n}$)? Because it breaks exactly where we need it most: near the edges 0 and 1, and at small $n$.

Take a real case from chapter 8: the guardrail let 0 dangerous requests through out of a 30-question test set.

from kobeval import wilson_ci      # the same function used since chapter 1

wilson_ci(0, 30)                   # → (0.000, 0.114)

• Normal approximation: $0 \pm 1.96\sqrt{0 \cdot 1/30} = 0 \pm 0$ — an interval of zero width, as if we were 100% certain the leak rate is 0% after watching just 30 examples
• Wilson: $[0\%,\ 11.4\%]$ — "in the 30 tries we've watched, nothing leaked, but the true rate could be as high as 11%"

The second interval is the honest sentence. The first is a lie the formula manufactures automatically. Notice in the formula how Wilson pulls the midpoint toward $1/2$ with the $z^2/2n$ term (like adding $z^2 \approx 4$ phantom questions, half right, half wrong) and how the $z^2/4n^2$ term keeps the width from collapsing to zero — this is why kobeval has used Wilson all series long.

3.2 Length-normalised multiple-choice scoring — the number that quietly decides leaderboards​

Log-likelihood mode has the model read the question $q$ and compare the probability of each full choice $c_1,\dots,c_m$:

• $\gamma = 0$ → raw total log-prob, which is biased toward short choices (fewer tokens = fewer probability multiplications)
• $\gamma = 1$ → divide by token count, i.e. mean log-prob per token

These two lines are acc and acc_norm in lm-evaluation-harness, and on real benchmarks they give different scores and sometimes reorder the models — when two papers report different ThaiExam numbers, one of the top causes is a different $\gamma$, with neither paper ever writing down which value it used.

3.3 Unbiased pass@k — for problems that can be auto-checked​

Math and code problems let us sample several answers and ask "was any of them right?" Sample $n$ times, get $c$ correct; the correct estimator of pass@$k$ is

The intuitive estimator $1 - (1 - c/n)^k$ is biased: the function $f(p) = 1-(1-p)^k$ is concave in $p$, so by Jensen's inequality $\mathbb{E}[f(\hat p)] \le f(p)$ — plug an estimate into a nonlinear function and the expectation no longer matches the truth. The binomial formula above reads as "the fraction of $k$-subsets of the $n$ samples that are all wrong," and it can be proven exactly unbiased.

3.4 McNemar's test — comparing two models on the same exam​

Models A and B take the same exam. Let $b$ = questions A got right but B got wrong, $c$ = questions B got right but A got wrong:

Compare against $\chi^2_1$ (with continuity correction). The key word is paired: the questions both got right and the questions both got wrong appear nowhere in the formula, because they say nothing about who's better. If you compare two accuracies with a t-test as if they came from different exams, you throw away the "same question" structure and then need many times more questions for the same power.

3.5 Contamination check — did the exam leak into the training data​

For an exam question $x$ and a training corpus $\mathcal{D}$, define the $k$-gram overlap rate:

where $G_k(\cdot)$ is the set of all $k$-grams. For Thai we use character-level k-grams (e.g. 20 characters), because Thai word segmentation is ambiguous. If $\text{overlap}_k$ is high (say above 0.7), suspect the model has already "seen the answer key" — its score on that question measures memory, not ability.

4. Seeing the Equations​

CI width is a function of n — and n=100 gives ±10 points​

This is the most important figure of the chapter. The width shrinks as $1/\sqrt{n}$, so a 100-question test set can never separate models 4 points apart, no matter how many times you rerun it. And to read a 1-point gap with confidence, you need on the order of ten thousand questions — which most Thai benchmarks don't have.

What a leaderboard looks like once you actually draw the error bars​

Models B, C, D differ by at most 5 points, but the CIs are ±9 points wide — the only conclusion the data supports is "these three cannot be told apart." Anyone declaring that D beats B is reading signal out of noise.

What people don't report is bigger than what people argue about​

McNemar: 100 questions, but only 10 carry information​

The two models sit 8 points apart (79% versus 71%) — sounds decisive, but the actual evidence is the 10 questions where they disagree. McNemar says this only just crosses the $p = 0.05$ line. With $b=6, c=4$ (a 2-point gap, like a typical leaderboard), you'd get $\chi^2 = 0.1$ — not remotely significant.

Play with the n → CI relationship yourself​

This widget is the same one from chapter 8 (guardrails), but this time look at it from a different angle: every TPR / FPR / precision number in it carries the same Wilson CI as equation 3.1. Drag the threshold $\tau$ toward an extreme until one cell of the confusion matrix holds only a few examples, and watch the CI balloon before your eyes — that is figure 9.1 in interactive form.

5. Setting Up the Environment​

Open Colab and pick Runtime → Change runtime type → T4 GPU (the free tier is enough). This chapter is pure inference — no optimizer, no gradients — sweeping every 0.6B-sized checkpoint takes about 15 minutes in total, more relaxed than any chapter so far.

torch_dtype=torch.float16      # not bfloat16
attn_implementation="sdpa"     # not flash_attention_2

cap = torch.cuda.get_device_capability(0)
print("compute capability:", cap)                    # T4 = (7, 5)
print("native bf16:", cap[0] >= 8)                   # T4 -> False
print("torch says   :", torch.cuda.is_bf16_supported())  # T4 -> True (counts emulation!)

Every checkpoint in the series builds on the same Qwen3-0.6B base, and most are LoRA adapters, so we load the base weights once and swap adapters in and out one at a time — VRAM doesn't grow with the checkpoint count:

CHECKPOINTS = {
    "base":        None,                                   # plain Qwen3-0.6B-Base
    "01-cpt":      "kobkrit/qwen3-0.6b-th-cpt",            # full weights (chapter 1)
    "02-sft":      "kobkrit/qwen3-0.6b-th-sft-lora",       # LoRA (chapter 2)
    "03-ppo":      "kobkrit/qwen3-0.6b-th-ppo-lora",       # LoRA (chapter 3)
    "04-dpo":      "kobkrit/qwen3-0.6b-th-dpo-lora",       # LoRA (chapter 4)
    "05-grpo":     "kobkrit/qwen3-0.6b-th-grpo-lora",      # LoRA (chapter 5)
    "06-ctx-dist": "kobkrit/qwen3-0.6b-th-ctxdist-lora",   # LoRA (chapter 6)
    "07-distill":  "kobkrit/qwen3-0.6b-th-distill",        # the student from chapter 7
    "08-guarded":  "kobkrit/qwen3-0.6b-th-sft-lora+guard", # model + filter from chapter 8
}

6. Preparing the Data​

6.1 scb10x/thai_exam — real Thai standardized exams​

A multiple-choice set drawn from Thailand's actual national exams (O-NET, IC, TGAT, TPAT-1, A-Level — the university-entrance and professional-certification exams Thai students sit). This is the Thai benchmark public leaderboards use most, and it is the backbone of this chapter.

from huggingface_hub import DatasetCard

card = DatasetCard.load("scb10x/thai_exam")
print("license:", card.data.get("license"))
print(card.text[:1500])        # read the card's terms with your own eyes

6.2 VISAI-AI/gsm8k-thai — generative math problems​

GSM8K translated into Thai: math word problems where the model must produce the answer itself, with no choices to compare log-likelihoods over — the dedicated proving ground for generative exact-match mode (check the license on the card here too).

6.3 KobEval-TH — the series' own 100-question set​

The benchmark used since chapter 1 (TH-KNOW Thai knowledge, instruction following, th_ratio — the fraction of Thai characters in a model's output, this series' standing metric for catching silent drift into English). Its virtue is not size — 100 questions gives ±10-point CIs, per figure 9.1 — but constancy: every chapter measures with the same set, the same way, so cross-chapter comparison is real.

6.4 Thai normalization — where Thai exact-match dies most often​

Thai has its own digit characters (๐๑๒๓๔๕๖๗๘๙ for 0–9), still common in official documents. So "๕๐", "50", and " 50 " are all the same answer — but Python's == doesn't think so:

import re

THAI_DIGITS = str.maketrans("๐๑๒๓๔๕๖๗๘๙", "0123456789")

def normalize_thai(s: str) -> str:
    s = s.strip().translate(THAI_DIGITS)   # Thai digits ๐-๙ → Arabic
    s = s.replace(",", "")                 # 1,000 → 1000
    s = re.sub(r"\s+", "", s)              # Thai doesn't separate words with spaces — drop them all
    return s

The notebook ships a small unit test for this function, because a bug in the grader is contamination in reverse: it makes the model look systematically worse than it is, and no error will ever tell you.

7. The Main Code​

7.1 Mode 1 — log-likelihood multiple-choice (equation 3.2, literally)​

import torch

@torch.no_grad()
def score_mc(model, tok, question, choices, gamma=1.0):
    scores = []
    for c in choices:
        q_ids    = tok(question, return_tensors="pt").input_ids.cuda()
        full_ids = tok(question + c, return_tensors="pt").input_ids.cuda()
        logits   = model(full_ids).logits[:, :-1]
        targets  = full_ids[:, 1:]
        logp = torch.log_softmax(logits.float(), -1).gather(
            -1, targets.unsqueeze(-1)).squeeze(-1)
        ans = logp[0, q_ids.shape[1] - 1:]            # the choice's tokens only
        scores.append(ans.sum().item() / len(ans) ** gamma)
    return int(torch.tensor(scores).argmax())         # not a single token is generated

Strengths: 100% deterministic, fast, works even on base models that can't yet answer in sentences. Limits: multiple-choice only, and the score depends on $\gamma$, as section 9 shows.

7.2 Mode 2 — generative exact-match​

@torch.no_grad()
def score_generative(model, tok, question, gold):
    msgs = [{"role": "user", "content": question}]
    ids = tok.apply_chat_template(msgs, add_generation_prompt=True,
                                  enable_thinking=False,        # the series' standing contract
                                  return_tensors="pt").cuda()
    out = model.generate(ids, max_new_tokens=256, do_sample=False)  # greedy
    pred = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return int(normalize_thai(extract_answer(pred)) == normalize_thai(gold))

extract_answer pulls the final answer out of the text (the last number for GSM8K-TH, the choice letter for multiple-choice) — this function is itself a score-affecting decision, and must be reported.

7.3 Mode 3 — LLM-as-judge with a written rubric​

The notebook's rubric is written in Thai, since it grades Thai answers; in English it reads:

JUDGE_RUBRIC = """You are an exam grader. Judge by this rubric only:
1 = the substance matches the gold answer (accept spelling variants,
    Thai/Arabic numerals, and different phrasings with the same meaning)
0 = wrong, off-topic, or no answer
Do not award points for elegant language. Do not award points for length.
Answer in JSON only: {"score": 0 or 1, "reason": "brief"}

Question: {q}
Gold answer: {gold}
Model's answer: {pred}"""

The judge must be a model far stronger than the one being judged. The notebook is designed to plug in any OpenAI-compatible endpoint, and it always records the rubric + the judge's model name into results.json — judge results that don't say who the judge was and what rubric it used are just another kind of rumor.

7.4 What the professionals use — lm-evaluation-harness​

We wrote the three modes ourselves to see the internals, but real work should stand on tooling the community has vetted:

pip install lm-eval

lm_eval --tasks list | grep -i thai        # first check which task names actually exist in your version

lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen3-0.6B,dtype=float16 \
    --tasks thai_exam \
    --num_fewshot 5 \
    --batch_size 8 \
    --seed 42 \
    --log_samples --output_path results/harness

--log_samples is the most important flag on those lines: it records per-question answers, which lets us (1) compute Wilson CIs ourselves, (2) pair questions for McNemar, and (3) trace exactly why each wrong answer was wrong. Notice that the harness reports both acc and acc_norm — those are $\gamma=0$ and the length-normalised form of equation 3.2.

8. Results​

The whole-series summary figure — every checkpoint on one axis, with error bars​

The notebook ends with the figure this series has walked 8 chapters to draw: every checkpoint from chapters 1–8 on a single axis, measured under a single contract, with a Wilson 95% CI on every bar (the real numbers live in the notebook's results.json — so this table stays ? until you run it yourself):

What to expect before running (always write the hypothesis down before looking — the best habit this chapter can teach): most checkpoints' CIs will overlap on ThaiExam, because our training runs are minuscule next to pretraining. The differences that should survive their error bars are th_ratio (chapter 4 attacked it head-on) and instruction-format compliance on KobEval-TH (the work of chapter 2's SFT). If the real figure comes out differently, that is something to be excited about, not something to hide.

One answer, three judges, three scores​

The samples above are real model answers where the same question earns different scores under the three modes — for instance, answering "๕๐ บาท" ("50 baht", with the amount in Thai numerals): exact-match without normalization gives 0, with normalization gives 1, and the judge gives 1. Or an answer whose reasoning is right all the way but botches the final number, which the judge sometimes grades more kindly than the answer key.

The honesty exercise: reproduce a leaderboard number — or find out exactly why you can't​

The notebook closes with the task that teaches more than every other cell combined: open a public leaderboard that reports Qwen3-0.6B on ThaiExam, write down the number they announce, then try to produce that same number from your own machine.

If it doesn't match (and on the first attempt it usually doesn't), do not stop at "close enough" — walk the variables one at a time:

1. The prompt template — does it match theirs? (figure 9.3 already told you this alone is more than enough)
2. The scoring mode — did they use log-likelihood or generative, acc or acc_norm ($\gamma$!)
3. Shot count — 0-shot and 5-shot are different worlds
4. enable_thinking — Qwen3 has a private thinking mode; on/off changes both score and runtime
5. Dataset version and subset — thai_exam has five subjects; how did they average?
6. Batch size during generation — different padding genuinely can change greedy outputs

A report saying "we got 41.8 while the leaderboard reports 43.5, and the cause is that they used 5-shot with acc_norm while we used 0-shot with acc" is worth more than a report whose number matches exactly but can't explain why — because the first proves you control your own measuring instrument. The second may just be lucky.

9. Comparison​

Every previous chapter compared "many models, one measurement." This chapter flips it: one model (the SFT checkpoint from chapter 2), many measurements, watching how far the score swings under decisions that normally go unreported:

(GSM8K-TH has no choices to compare log-likelihoods over, so only generative mode applies — that cell is blank on purpose.)

Every row is the same model, the same weights to the byte. What you should see is the score swinging by several points — more than the gap between models on a typical leaderboard. And that is the final answer to the question this chapter opened with: when different groups report numbers that disagree, most of the time nobody is lying — they just never measured with the same instrument.

Traps to watch for​

1. Contamination — the exam leaked into our own training data The notebook runs the equation 3.5 check (20-char-grams) between the ThaiExam / KobEval-TH questions and the corpora we trained on in chapters 1–2 (thaigov-v2 and the synthetic instruct set). The place to watch: thaigov is Thai government documents, and ThaiExam includes questions on government rules, regulations, and civic knowledge — real overlap is plausible. If you find hits, report them per-question and exclude them from the summary, don't look away. (The actual scan output prints in the notebook — the hit count is ? until you run it.)

2. Comparing numbers across papers with different templates "Our model got 45 on ThaiExam and that paper reports 43" means nothing at all if the template, the shot count, and $\gamma$ differ — only numbers measured under the same harness, same config are comparable.

3. Giving one model the chat template and not the other Scoring a base model through a chat template = handicapping it for free. Scoring an instruct model without its template = the same handicap. The section 8 table contains both kinds (CPT is base, the rest are chat), so the notebook prints the actual first prompt for every model for you to eyeball — the one line that keeps the whole table fair.

4. The judge favors its own family (self-enhancement bias) Multiple studies find LLM judges score answers written in their own family's style higher than they should. If the judge and the judged are relatives, the numbers come out suspiciously sweet — the correct fix isn't to find a "neutral" judge (there isn't one) but to always report the judge's name, and cross-check with a judge from another family when results are close.

5. Thai exact-match breaking on Thai numerals The model answers "๕๐", the answer key says "50" — instant zero if you forgot normalize_thai. And this bug doesn't spread evenly: a model CPT-trained on government documents (which use Thai numerals heavily) loses more points than its peers. It becomes a systematic bias that picks on specific models — the grader must be fair before anyone talks about rankings.

10. Summary​

• Accuracy without a CI is a rumor — n=100 gives ±10 points, and most leaderboard gaps are smaller than that
• Wilson is not a luxury — the normal approximation gives a zero-width CI at $\hat p = 0$, exactly where we need it most
• A score is a property of (model × method × exam × n) — scoring mode, $\gamma$, shots, and template can move a score more than models actually differ
• Two models on the same exam: use McNemar — the questions they agree on are not evidence
• pass@k needs the binomial estimator — the plug-in estimator is biased, by Jensen
• Scan for contamination before believing a score, especially when training data and exams come from neighboring domains
• Declare a measurement contract on day one and never touch it again — this series': greedy, 256 tokens, thinking off, seed 42
• Reproducing someone's number and explaining the difference beats matching it without knowing why

Next chapter: Deployment — a model that has been measured must go out and meet real users. What quantization trades away, what machines can serve it, and how this chapter's numbers become production regression tests.

References​

1. Liang et al. (2022). Holistic Evaluation of Language Models — HELM: multi-dimensional evaluation instead of one number
2. Biderman et al. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models — lessons from lm-evaluation-harness on reproducible evaluation
3. Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — why every accuracy number needs an error bar
4. Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — LLM-as-a-judge and its biases
5. Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — ranking by real human preference
6. Hendrycks et al. (2020). Measuring Massive Multitask Language Understanding — MMLU: the template for multiple-choice benchmarks
7. Chen et al. (2021). Evaluating Large Language Models Trained on Code — the unbiased pass@k estimator used in section 9
8. Wilson (1927). Probable Inference, the Law of Succession, and Statistical Inference — the Wilson interval used on every number in this series
9. McNemar (1947). Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages — the paired test for two models on the same items

The writing, code and notebooks in this series are licensed under CC BY-NC-SA 4.0 — reuse and adapt them freely with attribution, for non-commercial purposes, and share your adaptations under the same terms. Third-party models and datasets referenced here keep their own licences.