[LLM 1/10] Continue Pretraining: Teaching New Knowledge to a Thai LLM

A large language model that handles Thai reasonably well will still, more often than not, know nothing about your organization's specialized knowledge — not Thai government regulations, not the jargon of your industry, not your internal documents. This article covers the most direct fix: Continue Pretraining (CPT), from the equations all the way to code that runs end to end on free Colab in about 15 minutes.

1. The Problem​

Picture asking Qwen3-0.6B something like "Under the Office of the Prime Minister's regulations, when is procurement by the specific-method route permitted?" The model will answer confidently, and it will be wrong, because it has never seen enough Thai government documents.

Plenty of people try to fix this with fine-tuning on a few thousand question-answer pairs, and find it doesn't work. The reason is that SFT teaches the shape of an answer, not the knowledge behind it. If the knowledge isn't in the weights, teaching the model to answer in the right tone of voice just makes it confident while it lies.

New knowledge reaches a model through three routes, and picking the wrong one is why most LLM projects fail:

This article is about the second route.

2. What We're Going to Do​

We take a base model (one that hasn't been through instruction tuning) and keep training it with the exact same objective used during pretraining — next-token prediction — on raw Thai text from the domain we care about. No labels, no question-answer pairs, just plain text.

But the heart of this article isn't "train it and it gets better." It's what you give up in exchange:

The phenomenon of a model forgetting what it used to be able to do is called catastrophic forgetting. We won't just gesture at it — we'll measure it as a number and then find a point we can live with.

3. The Equations​

3.1 The CPT objective​

• $x_t$ = the token at position $t$
• $x_{<t}$ = every token before it
• $p_\theta$ = the probability the model predicts

This equation is identical to the one used during pretraining. The only thing that changes is the data. That's why CPT needs no labels — the text is its own answer key.

3.2 Perplexity: our unit of measurement​

In plain language: "on average, how many options is the model torn between?" PPL = 20 means it's wavering among roughly 20 candidates; PPL = 5 means it's far more certain. Lower is better.

3.3 The most important equation in this chapter — replay mixing​

$\lambda$ is the share of domain data in each batch.

• $\lambda = 1.0$ → pure domain data → fastest domain gains and the fastest forgetting
• $\lambda = 0.5$ → an even mix → slower, but far less forgetting

Don't hardcode this value. Sweep it, then pick the point you can accept.

4. Seeing the Equations​

What perplexity actually tells us​

The right-hand plot shows what people usually get wrong: if someone tells you they "cut perplexity by 5 points" without saying where they started, the sentence is nearly meaningless. 80 → 75 is a 6% improvement; 10 → 5 is 50%.

The trade-off the replay ratio controls​

5. Setting Up the Environment​

Open Colab and pick Runtime → Change runtime type → T4 GPU (the free tier is enough).

torch_dtype=torch.float16      # not bfloat16
attn_implementation="sdpa"     # not flash_attention_2
fp16=True                      # in TrainingArguments (not bf16=True)

The first cell of every notebook in this series prints this line so you can see it with your own eyes:

cap = torch.cuda.get_device_capability(0)
print("compute capability:", cap)                    # T4 = (7, 5)
print("native bf16:", cap[0] >= 8)                   # T4 -> False
print("torch says   :", torch.cuda.is_bf16_supported())  # T4 -> True (counts emulation!)

How much VRAM you have, and where it goes​

Notice that the optimizer state takes up more room than the model itself — Adam keeps one copy each of $m$ and $v$ for every parameter. Switching to adamw_bnb_8bit saves 3.3 GB, which means a lot more headroom for batch size or sequence length.

Play with the VRAM budget yourself — change the numbers and see where it OOMs:

6. Preparing the Data​

We use pythainlp/thaigov-v2-corpus-22032023 — a corpus of Thai government news and official documents (public domain). It stands in for "specialized knowledge the model has never seen enough of."

We also use a second set of general Thai text as replay data to hold back the forgetting.

from datasets import load_dataset

domain = load_dataset("pythainlp/thaigov-v2-corpus-22032023", split="train")
domain = domain.shuffle(seed=42).select(range(8000))

Packing: don't let padding eat your budget​

If you pad every document out to the same length, you burn an enormous amount of compute on <pad>. The right approach is to concatenate every document and slice the stream into equal 512-token blocks.

def pack(examples, block_size=512):
    ids = []
    for text in examples["context"]:
        ids.extend(tokenizer(text + tokenizer.eos_token).input_ids)
    n = (len(ids) // block_size) * block_size
    return {"input_ids": [ids[i:i+block_size] for i in range(0, n, block_size)]}

7. The Main Code​

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B-Base",           # base, not instruct — CPT must start from a base model
    torch_dtype=torch.float16,        # the T4 has no bf16
    attn_implementation="sdpa",       # the T4 has no FlashAttention-2
).cuda()

model = model.float()                 # must cast to fp32 before training — see the box below

args = TrainingArguments(
    output_dir="cpt-out",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,    # effective batch = 16
    num_train_epochs=1,
    learning_rate=2e-5,               # 10x lower than SFT — see the warning below
    lr_scheduler_type="cosine",
    warmup_steps=50,
    optim="adamw_bnb_8bit",           # saves 3.3 GB
    gradient_checkpointing=True,
    max_grad_norm=1.0,                # keeps fp16 from blowing up
    fp16=True,                        # not bf16
    logging_steps=10,
)

ValueError: Attempting to unscale FP16 gradients.

8. Results​

The notebook measures three things before and after training and writes them to results.json:

1. Domain held-out PPL — should drop clearly (this is what we're paying for)
2. General held-out PPL — should rise somewhat (this is the price)
3. TH-KNOW accuracy from the KobEval-TH benchmark, with a Wilson 95% CI

9. Comparison​

The notebook trains three variants on the same data so you can see the trade-off in numbers:

Reading this table correctly is the point of the chapter:

• λ = 1.0 wins on domain PPL (4.07, the lowest) but general PPL gets worse, 5.88 → 6.72. That is catastrophic forgetting, measured rather than asserted.
• λ = 0.5 gives up a little domain accuracy (4.29) and general perplexity improves to 4.98. Replay does not merely prevent forgetting here; it leaves the model better at Thai overall.

10. Summary​

• CPT puts knowledge into the weights using the same objective as pretraining, with no labels required
• It is always a trade — domain accuracy is bought with general capability you lose
• The replay ratio $\lambda$ is the dial that sets the exchange rate — sweep it, don't guess
• A low learning rate is the line between doing CPT and destroying the model
• Every number needs a confidence interval

Next chapter: SFT and LoRA — once the model has the knowledge, how do we teach it to answer, and why does training just 1.7% of the parameters get you almost as far as training all of them?

References​

1. Gururangan et al. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks — the domain-adaptive pretraining recipe this chapter follows
2. Ibrahim et al. (2024). Simple and Scalable Strategies to Continually Pre-train Large Language Models — the replay and LR strategies that keep CPT from destroying the model
3. Gupta et al. (2023). Continual Pre-Training of Large Language Models: How to (re)warm your model? — why LR warmup matters so much when continuing pretraining
4. Luo et al. (2023). An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning — a systematic measurement of catastrophic forgetting
5. Kaplan et al. (2020). Scaling Laws for Neural Language Models — scaling laws -- the basis for calling 8,000 documents far too little
6. Hoffmann et al. (2022). Training Compute-Optimal Large Language Models — Chinchilla: the compute-optimal data-to-parameter ratio
7. Yuenyong et al. (2025). OpenThaiGPT 1.6 and R1: Thai-Centric Open Source and Reasoning Large Language Models — Thai CPT at real scale, for contrast with this chapter's toy run
8. Lowphansirikul et al. (2021). WangchanBERTa: Pretraining transformer-based Thai Language Models — the pioneering Thai LM and its corpus preparation

The writing, code and notebooks in this series are licensed under CC BY-NC-SA 4.0 — reuse and adapt them freely with attribution, for non-commercial purposes, and share your adaptations under the same terms. Third-party models and datasets referenced here keep their own licences.