[LLM 8/10] Guardrails: The Real Safety Fence Isn't a Model, It's a Threshold

The Thai chatbots my team and I deploy for real customers don't only meet polite questions — people ask for recipes for illegal things, people try to trick the bot into insulting others, and there was a day the model volunteered a customer's phone number all by itself. In this chapter we build protection in both directions: a classifier that checks incoming prompts for danger and an outbound PII filter. But the central point of the chapter isn't the model — it's the fact that a guardrail is a threshold decision under asymmetric costs, and the "94% accuracy" figure people love to show off is very nearly meaningless.

Open in Colab08_guardrails.ipynb

Thai LLM seriesPart 8 of 10

1. 1Continue Pretraining
2. 2SFT and LoRA
3. 3RLHF and PPO
4. 4DPO: Direct Preference Optimization
5. 5GRPO
6. 6Context Distillation
7. 7Model Distillation
8. 8Guardrails(you are here)
9. 9Benchmarking
10. 10Deployment

1. The Problem

Every previous chapter trained the model to be "better." But no matter how capable a model is, the moment it meets real users, damage can always arrive from two directions:

Direction	Example damage	This chapter's tool
Inbound (input)	A user asks how to hurt someone, for an illegal recipe, how to cheat — and the model answers	a dangerous-prompt classifier
Outbound (output)	The model emits a national ID number, phone number, or bank account leaked from context or training data	a deterministic PII filter

Remember that SFT back in chapter 2 had a very quiet side effect: fine-tuning on narrow data erodes the refusal behavior the model used to have. A model you tuned yourself is therefore usually less safe than the original. That is why real systems need another fence that lives outside the model.

So why can't you just buy an off-the-shelf guardrail advertised as "94% accurate"? Because that sentence hasn't answered the three questions that matter most:

1. 94% accurate at which threshold — the same number can slide along the entire curve
2. Measured on a test set with what percentage unsafe — in production, genuinely dangerous traffic is usually under 1%
3. And what percentage of innocent users does it block — the number almost nobody agrees to report

A guardrail that blocks customers asking normal questions is not a safe system. It is a broken product.

2. What We're Going to Do

We'll build two real guardrails on free Colab, then measure them honestly:

Layer	Position	Technique	Approx. latency
Input guardrail	before the prompt reaches the LLM	Qwen3-0.6B + sequence-classification head + LoRA r=8	~15–30 ms
Output guardrail	after the LLM answers, before the user sees it	regex + mod-11 checksum (no ML at all)	~0.1 ms

The core idea of this chapter

A guardrail isn't a model — it's a threshold decision under asymmetric costs. The model only supplies a score $p_\phi(\text{unsafe}\mid x)$; choosing the cutoff $\tau$ is answering the business question "how many times more expensive is letting one dangerous thing through than blocking one innocent customer?"

An honest report therefore always carries two numbers: the unsafe caught and the benign blocked. A single number on its own is marketing, not engineering.

We'll also see that some of the best guardrails aren't ML at all — a PII filter built from regex + checksum is deterministic, unit-testable, microsecond-fast, and can never be jailbroken.

3. The Equations

3.1 The classifier loss — an old friend

$$\mathcal{L}(\phi) = -\mathbb{E}_{(x,y)\sim\mathcal{D}}\Big[\,y\log p_\phi(\text{unsafe}\mid x) + (1-y)\log\big(1-p_\phi(\text{unsafe}\mid x)\big)\Big]$$

Plain binary cross-entropy ($y=1$ means unsafe). Nothing new — and that is precisely the point: the ML part of a guardrail is the easiest part of the whole system. The real substance is below.

3.2 The decision rule and expected cost — the actual content of this chapter

$$\text{block}(x) = \mathbf{1}\big[\,p_\phi(\text{unsafe}\mid x) > \tau\,\big]$$

The model's job ends at producing a score. Blocking or passing is a comparison against $\tau$, which we choose by minimizing expected cost:

$$C(\tau) = c_{\text{FN}}\,P(\text{unsafe})\,\text{FNR}(\tau) \;+\; c_{\text{FP}}\,P(\text{safe})\,\text{FPR}(\tau) \qquad\qquad \tau^* = \arg\min_\tau C(\tau)$$

• $\text{FNR}(\tau)$ = the fraction of unsafe that slips through (false negative rate)
• $\text{FPR}(\tau)$ = the fraction of benign that gets blocked (false positive rate)
• $c_{\text{FN}}, c_{\text{FP}}$ = the price of each kind of mistake

Notice that this equation forces you to answer a question ML cannot answer for you: what is your product's $c_{\text{FN}}/c_{\text{FP}}$? This is a pure product decision, and different products answer it differently:

Product	Price of an FN (unsafe slips through)	Price of an FP (blocking an innocent)	Reasonable $c_{\text{FN}}/c_{\text{FP}}$
Health-advice chatbot	life-threatening + legal liability	mildly annoyed user	50:1 or more
Enterprise customer assistant	damaging headlines	more support tickets	~10:1
Internal employee tool	limited (users are identifiable staff)	daily workflow friction	~2:1

If you've never written this ratio down explicitly in a document, it means someone has been choosing $\tau$ for you by accident.

3.3 The most expensive lesson of the chapter: base rates can destroy precision

Suppose our classifier catches unsafe at TPR = 95% and wrongly blocks only FPR = 5% — sounds excellent. Question: of the messages it blocks, what percentage are actually unsafe? Straight Bayes, with $\pi = P(\text{unsafe})$ the fraction of unsafe in real traffic:

$$\text{precision} = P(\text{unsafe}\mid\text{block}) = \frac{P(\text{block}\mid\text{unsafe})\,\pi}{P(\text{block})} = \frac{\text{TPR}\cdot\pi}{\text{TPR}\cdot\pi + \text{FPR}\cdot(1-\pi)}$$

Plug in two scenarios:

• Balanced test set ($\pi = 0.5$): precision $= \dfrac{0.95 \times 0.5}{0.95 \times 0.5 + 0.05 \times 0.5} = 95\%$
• Real traffic ($\pi = 0.01$): precision $= \dfrac{0.95 \times 0.01}{0.95 \times 0.01 + 0.05 \times 0.99} \approx 16\%$

The exact same classifier — but in production, of every 6 blocked messages, 5 are innocent users. Because when unsafe is rare ($\pi$ small), the $\text{FPR}\cdot(1-\pi)$ term in the denominator swallows everything. This is why evaluating on a balanced set and then bragging "95% accurate" is self-deception.

3.4 Layered defence

Stack $K$ independent guardrails (blocklist → classifier → system prompt → random human review), where unsafe must fool every layer to get through, but benign gets blocked if any single layer trips:

$$\text{FNR}_{\text{sys}} = \prod_{k=1}^{K}\text{FNR}_k \qquad\qquad \text{FPR}_{\text{sys}} = 1 - \prod_{k=1}^{K}\big(1-\text{FPR}_k\big)$$

FNR falls geometrically (wonderful), but FPR compounds upward (the bill you pay).

The independence assumption is wildly optimistic

The $\prod_k \text{FNR}_k$ equation holds only if the layers fail independently, which in reality they almost never do — a single evasion trick (say, inserting a zero-width space mid-word) tends to fool every layer that works on raw text at once. The layers' errors are therefore correlated, and the real $\text{FNR}_{\text{sys}}$ is always worse than this formula. Treat it as a best case, not a promise.

3.5 Constrained decoding — the non-ML guardrail everyone overlooks

If your use case only ever answers from a fixed set (a menu, categories, schema-conforming JSON), don't inspect the text afterwards — enforce it at generation time by renormalizing over the allowed token set $\mathcal{A}$:

$$p'(t\mid x) = \frac{p_\theta(t\mid x)\,\mathbf{1}[t\in\mathcal{A}]}{\sum_{t'\in\mathcal{A}} p_\theta(t'\mid x)}$$

Tokens outside $\mathcal{A}$ have probability exactly zero, not "very small." The result is a guardrail that is deterministic, adds zero latency, and cannot be bypassed by any prompt in the universe — because it doesn't forbid the model from "wanting to say" something; it makes out-of-set words not exist in the inventory in the first place. Wherever constrained decoding applies, use it first, and save the classifier for the parts that are genuinely free text.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →