[LLM 1/10] Continue Pretraining: Teaching New Knowledge to a Thai LLM

A large language model that handles Thai reasonably well will still, more often than not, know nothing about your organization's specialized knowledge — not Thai government regulations, not the jargon of your industry, not your internal documents. This article covers the most direct fix: Continue Pretraining (CPT), from the equations all the way to code that runs end to end on free Colab in about 15 minutes.

Open in Colab01_continue_pretraining.ipynb

Thai LLM seriesPart 1 of 10

1. 1Continue Pretraining(you are here)
2. 2SFT and LoRA
3. 3RLHF and PPO
4. 4DPO: Direct Preference Optimization
5. 5GRPO
6. 6Context Distillation
7. 7Model Distillation
8. 8Guardrails
9. 9Benchmarking
10. 10Deployment

1. The Problem

Picture asking Qwen3-0.6B something like "Under the Office of the Prime Minister's regulations, when is procurement by the specific-method route permitted?" The model will answer confidently, and it will be wrong, because it has never seen enough Thai government documents.

Plenty of people try to fix this with fine-tuning on a few thousand question-answer pairs, and find it doesn't work. The reason is that SFT teaches the shape of an answer, not the knowledge behind it. If the knowledge isn't in the weights, teaching the model to answer in the right tone of voice just makes it confident while it lies.

New knowledge reaches a model through three routes, and picking the wrong one is why most LLM projects fail:

Approach	Best for	Cost at inference time
RAG	Knowledge that changes often and needs source citations	A retrieval on every call + a long prompt
Continue Pretraining	Large volumes of specialized knowledge that stay fairly stable	None (it's already in the weights)
SFT	Format, tone, answer structure	None

This article is about the second route.

2. What We're Going to Do

We take a base model (one that hasn't been through instruction tuning) and keep training it with the exact same objective used during pretraining — next-token prediction — on raw Thai text from the domain we care about. No labels, no question-answer pairs, just plain text.

But the heart of this article isn't "train it and it gets better." It's what you give up in exchange:

The core idea of this chapter

CPT buys domain accuracy by paying with general capability you lose along the way. It is a trade, not a free lunch, and the "exchange rate" is governed by a single number called the replay ratio.

The phenomenon of a model forgetting what it used to be able to do is called catastrophic forgetting. We won't just gesture at it — we'll measure it as a number and then find a point we can live with.

3. The Equations

3.1 The CPT objective

$$\mathcal{L}_{\text{CPT}}(\theta) = -\mathbb{E}_{x\sim\mathcal{D}_{\text{domain}}}\left[\sum_{t=1}^{|x|}\log p_\theta(x_t \mid x_{<t})\right]$$

• $x_t$ = the token at position $t$
• $x_{<t}$ = every token before it
• $p_\theta$ = the probability the model predicts

This equation is identical to the one used during pretraining. The only thing that changes is the data. That's why CPT needs no labels — the text is its own answer key.

3.2 Perplexity: our unit of measurement

$$\text{PPL}(\mathcal{D}) = \exp\!\left(\frac{1}{N}\sum_{i}\mathcal{L}_{\text{CPT}}(x^{(i)})\right)$$

In plain language: "on average, how many options is the model torn between?" PPL = 20 means it's wavering among roughly 20 candidates; PPL = 5 means it's far more certain. Lower is better.

3.3 The most important equation in this chapter — replay mixing

$$\mathcal{D}_{\text{mix}} = \lambda\,\mathcal{D}_{\text{domain}} + (1-\lambda)\,\mathcal{D}_{\text{general}}$$

$\lambda$ is the share of domain data in each batch.

• $\lambda = 1.0$ → pure domain data → fastest domain gains and the fastest forgetting
• $\lambda = 0.5$ → an even mix → slower, but far less forgetting

Don't hardcode this value. Sweep it, then pick the point you can accept.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →