[LLM 2/10] SFT + LoRA: Teaching a Model to Be an Assistant by Training 1.69% of Its Parameters

Last chapter we used Continue Pretraining to put knowledge into the model's weights. But a model that knows is not a model that answers — a base model has exactly one job: continuing text. This chapter teaches SFT (Supervised Fine-Tuning) with LoRA: a technique that trains only about 1.7% of the parameters yet changes the behavior of the entire model — finished in ~15 minutes on free Colab. What you get back is an adapter file of roughly 40 MB that becomes the backbone of the rest of the series.

1. The Problem​

Take a true base model like Qwen3-0.6B-Base from the last chapter and type in (in Thai) "Could you recommend some Thai dishes, please?" What comes back is usually not an answer but a continuation — it may compose three more questions, keep going as a travel article, or switch into English partway through, because the only thing it was ever trained on is "on the internet, what usually follows text like this?"

The ability to answer — take an instruction, respond to the point, then stop — does not come with pretraining. It comes from SFT: continued training on (instruction, good answer) pairs, thousands to millions of them. Every instruct model you have ever used has been through this stage.

But the moment you try it yourself, you hit two problems stacked on top of each other:

Layer one — the cost of full fine-tuning. Train every parameter and you get an entire new model (~1.2 GB per task for a 0.6B model). An organization with ten tasks — document summaries, letter drafting, customer replies, complaint triage — has to store ten copies. And moving every single weight with a high learning rate is a recipe for erasing the knowledge you just added in chapter 1 (remember the learning-rate box?).

Layer two — Thai. Even the post-trained Qwen3-0.6B shows the symptom we will see all series long: ask in Thai, and the answer drifts into English mid-sentence (this is where the series' th_ratio metric comes from), because the SFT data it saw was overwhelmingly English.

This chapter fixes both layers at once: SFT on Thai instruction data, done through LoRA instead of full fine-tuning.

2. What We're Going to Do​

We take Qwen3-0.6B and train it on 4,000 Thai instruction-answer pairs, using the exact same loss as chapter 1 plus two new pieces:

1. Completion mask — compute the loss only on the answer tokens, never the question (section 3.1 explains the comically broken failure you get if you skip this)
2. LoRA (Low-Rank Adaptation) — freeze all the original weights and instead train two small matrices laid over each layer

3. The Equations​

3.1 The SFT loss and the completion mask​

• $(x, y)$ = one training pair — $x$ is the instruction part (chat template included) and $y$ is the token sequence of the example the model actually sees during training
• $y_t$ = the token at position $t$, and $y_{<t}$ = every token before it
• $p_\theta$ = the probability predicted by the model with parameters $\theta$
• $m_t \in \{0,1\}$ = the completion mask — 1 only on answer-side tokens, and 0 on prompt tokens

Remove $m_t$ (i.e. set $m_t \equiv 1$ everywhere) and this equation instantly becomes the CPT objective from chapter 1. SFT is CPT on text staged as a conversation, plus one mask — nothing more.

But that one mask is half the battle, because $m_t \equiv 1$ is the easiest default to fall into (many pipelines, SFTTrainer included, will silently train this way if the collator isn't wired correctly). And in typical Thai instruction data, prompt-side tokens make up around 60% of each example — meaning most of your gradient is busy teaching the model to write the user's questions, not to answer them. The side effect this causes will make its appearance in section 9.

3.2 LoRA: training the correction, not the weights​

Instead of updating the weight matrix $W_0$ directly, LoRA pins $W_0$ in place and learns a difference that is the product of two small matrices:

• $W_0 \in \mathbb{R}^{d\times k}$ = the layer's original weights, frozen — they receive no gradient at all
• $B \in \mathbb{R}^{d\times r}$ and $A \in \mathbb{R}^{r\times k}$ = the two adapter matrices we actually train
• $r \ll \min(d, k)$ = the rank of the correction — LoRA's main dial (this chapter uses $r = 16$)
• $\alpha$ = a scale multiplier — the product $BA$ is always multiplied by $\alpha/r$ (here $\alpha = 32$, so $\alpha/r = 2$)

Two details in this definition matter more than they look:

$B$ is initialized to all zeros, so $\Delta W = 0$ at the first step — training starts from exactly the base model, with no period where random weights disturb it. ($A$ is random Gaussian — set both to zero and both gradients stay zero forever, since each is multiplied by the other's zeros.)

The divisor $r$ in $\alpha/r$ makes the update magnitude independent of the rank — double $r$ and the sum $BA$ has twice as many terms, but it gets divided right back. So you can sweep $r$ without re-tuning the learning rate every time.

And when training finishes, you have two choices: merge ($W' = W_0 + \frac{\alpha}{r}BA$, giving a single model with no added latency) or keep it separate — the second is the road this series takes, because a detachable adapter is what chapter 4 uses to build a reference model for free.

3.3 The fraction of parameters trained — a number to check, not to recite​

For a single $d \times k$ matrix, the adapter has $rd + rk$ parameters, a fraction of

• $d, k$ = the dimensions of the original weight matrix
• $r$ = the adapter's rank

Plug in the real values for Qwen3-0.6B (hidden 1024, intermediate 3072, 28 layers, adapters on all 7 matrices: q, k, v, o, gate, up, down) and you get 10,092,544 trainable parameters on a base of 596,049,920 = 1.69%.

4. Seeing the Equations​

One mask redirects the entire gradient​

There is nothing deeper in this figure than counting tokens — but the counting is exactly what people skip: a Thai prompt (system message and chat template included) is often longer than its answer, so training unmasked spends most of your compute teaching something you never wanted. The notebook also prints the actual ratio for the sampled dataset — the 60/40 in the figure stands in for a typical example; it is not a sacred constant.

Why rank 16 is "enough" — the low-rank hypothesis​

LoRA's hypothesis (Hu et al., 2021) is that fine-tuning moves the weights along only a few important directions relative to the full dimensionality of the matrix. This figure builds a fake $\Delta W$ with exactly that structure (a rank-16 signal + noise) and lets SVD dig it back out — if real updates look like this, a rank-16 adapter captures nearly all of the content. What the figure does not prove is that real updates always look like this. That is an empirical finding of the research, and it is why LoRA "usually" comes close to full fine-tuning — not "always."

Check the number, don't recite it​

A straight line on log-log axes confirms what the equation says: the fraction grows exactly linearly with $r$. And on this model, "under 1%" is only true when $r \le 9$ — which is not a value anyone actually uses for language tasks. The point we use ($r=16$) is 1.69%, or about 40 MB stored as fp32.

5. Setting Up the Environment​

Open Colab and pick Runtime → Change runtime type → T4 GPU (the free tier is enough).

torch_dtype=torch.float16      # not bfloat16
attn_implementation="sdpa"     # not flash_attention_2
fp16=True                      # in SFTConfig (not bf16=True)

cap = torch.cuda.get_device_capability(0)
print("compute capability:", cap)                    # T4 = (7, 5)
print("native bf16:", cap[0] >= 8)                   # T4 -> False
print("torch says   :", torch.cuda.is_bf16_supported())  # T4 -> True (counts emulation!)

for p in model.parameters():
    if p.requires_grad:
        p.data = p.data.float()      # cast the adapter only — not the whole model

The VRAM budget LoRA transforms​

Remember how in chapter 1 Adam's optimizer state took more room than the model itself (about 4.4 GB for $m$ and $v$)? Training only the adapter, Adam keeps state for just 10.1 million parameters — about 0.08 GB — to the point that adamw_bnb_8bit is no longer needed. The big remaining budget items are the base weights (fp16, ~1.2 GB) and activations.

Try switching between full fine-tuning and various LoRA ranks in the calculator and watch where the budget moves:

6. Preparing the Data​

We use airesearch/wangchanx-seed-free-synthetic-instruct-thai-120k — 120,000 synthetic Thai instruction-answer pairs from the WangchanX team, a Thai open-source NLP research effort (we sample 4,000).

from datasets import load_dataset

ds = load_dataset("airesearch/wangchanx-seed-free-synthetic-instruct-thai-120k", split="train")
ds = ds.shuffle(seed=42).select(range(4000))

def to_text(ex):
    messages = [
        {"role": "user", "content": ex["instruction"]},   # always check column names against the dataset card
        {"role": "assistant", "content": ex["output"]},
    ]
    return {"text": tok.apply_chat_template(messages, tokenize=False,
                                            enable_thinking=False)}

train_ds = ds.map(to_text, remove_columns=ds.column_names)

Two lines worth reading slowly:

• apply_chat_template assembles the messages into the format Qwen3 was trained on (<|im_start|>user … <|im_end|> …). We call it once, here and nowhere else — the reason lives in trap 2 of section 9.
• enable_thinking=False turns off Qwen3's thinking mode, keeping this chapter's examples simple and the mask straightforward.

Then perform the ritual that should become a habit: always decode the first example and look at it with your own eyes.

print(train_ds[0]["text"][:400])
# you must see <|im_start|>user ... <|im_end|> ... <|im_start|>assistant ... exactly once each per turn

The notebook also prints token-length statistics for the sampled set: the median on the prompt side versus the answer side, and the fraction of examples longer than 768 tokens — that last number will come back to collect its debt in trap 3.

7. The Main Code​

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, SFTConfig, DataCollatorForCompletionOnlyLM

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",                # the post-trained model — this chapter fixes behavior, not knowledge
    torch_dtype=torch.float16,        # the T4 has no bf16
    attn_implementation="sdpa",       # the T4 has no FlashAttention-2
).cuda()

lora = LoraConfig(
    r=16,
    lora_alpha=32,                    # α/r = 2 — see equation 3.2
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()    # trust this line, not the number in any blog

for p in model.parameters():          # the fp16 box from section 5
    if p.requires_grad:
        p.data = p.data.float()

Next comes the piece that turns equation 3.1 into code — the collator that plays the role of $m_t$:

collator = DataCollatorForCompletionOnlyLM(
    response_template="<|im_start|>assistant\n",   # everything before this = prompt
    tokenizer=tok,
)

This collator sets the label of every token before <|im_start|>assistant to -100, the value PyTorch's loss function skips — the same value chapter 4 uses to mask out the prompt side in its seq_logp function. Same mask, showing up in a different chapter.

cfg = SFTConfig(
    output_dir="sft-out",
    dataset_text_field="text",
    max_seq_length=768,               # the notebook prints how many examples get truncated — trap 3
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,    # effective batch = 16
    num_train_epochs=1,
    learning_rate=2e-4,               # fine for LoRA — catastrophic for full FT (chapter 1)
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    packing=False,                    # the completion mask can't be used with packing directly
    fp16=True,                        # T4: not bf16
    logging_steps=10,
)

trainer = SFTTrainer(
    model=model,
    args=cfg,
    train_dataset=train_ds,
    data_collator=collator,
    processing_class=tok,
)
trainer.train()                       # ~15 minutes on a T4 (measured: 14.8)

Training done, save only the correction:

model.save_pretrained("qwen3-0.6b-th-sft-lora")   # ~40 MB — not 1.2 GB

This is the very adapter chapter 4 loads under the name kobkrit/qwen3-0.6b-th-sft-lora, serving as both the starting policy and (with the adapter switched off) the reference model — a cross-chapter gift priced at 40 MB.

8. Results​

The notebook measures 3 things before and after training and writes them to results.json:

1. TH-INSTR — the fraction of answers passing an instruction-following rubric (answers the question, stops on its own, stays in role) from the KobEval-TH benchmark, with a Wilson 95% CI
2. th_ratio — the fraction of Thai characters in the answers, the series' standing metric. The stock Qwen3-0.6B is notorious for drifting into English on Thai prompts; SFT on 4,000 all-Thai examples should move this number visibly — and if it doesn't, that is your signal to go hunt through the mask and the template before anything else.
3. The fraction of answers ending with eos within the token budget — the detector for the "won't stop" symptom from traps 1 and 3.

9. Comparison​

The notebook trains 3 variants on the same data, measured against the starting model:

SFT improves on every axis: TH-INSTR +10 points (73.3% → 83.3%), th_ratio up (more answers stay in Thai instead of drifting to English), and perplexity down from 21.3 to 17.7 — training only 1.69% of the parameters for a ~40 MB adapter.

On full fine-tuning and the no-mask row: the notebook does train a no-mask LoRA to show the behaviour where the model finishes its answer and then invents the user's next question (samples in §8), but it does not score full metrics for it, and it does not run a 100%-parameter full fine-tune — that exceeds the free-Colab budget. We report only what was measured and do not fill in numbers for runs we did not do.

10. Summary​

• SFT is CPT on conversations + a completion mask — the same loss as chapter 1; what changes is that the data is staged, and the mask chooses which tokens count
• The mask is half the battle — $m_t \equiv 1$ sends 60% of the gradient into practicing questions, and yields a model that answers, then writes the next question itself
• LoRA trains a "correction," not the weights: $W' = W_0 + \frac{\alpha}{r}BA$ with $B$ starting at zero, so training departs from exactly the base — and $\alpha/r$ lets you change rank without re-tuning the lr
• 1.69% is not "under 1%" — adapters grow linearly with hidden size while the base grows quadratically, so the smaller the model, the bigger the adapter's share; always check with print_trainable_parameters()
• ~40 MB of adapter per task: one base plus many adapters — and it is the reason chapter 4's reference model comes free
• lr 2e-4 is safe because the base is frozen — the same value destroys the model when training everything
• pad ≠ eos, template once, never truncate mid-answer — three traps whose symptoms surface at inference but whose causes live in the data

Next chapter: RLHF and PPO — the model can answer now, but "answers well" cannot be written as a loss function. We will have humans compare answers, train a reward model from those preferences, then use reinforcement learning to push the model toward it — with this chapter's 40 MB adapter as the starting point.

References​

1. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models — the source of the W' = W₀ + (α/r)BA equation in section 3
2. Aghajanyan et al. (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — evidence that fine-tuning has low intrinsic dimension -- why LoRA works at all
3. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs — extends LoRA with 4-bit so larger models fit one GPU
4. Biderman et al. (2024). LoRA Learns Less and Forgets Less — LoRA trades less learning for less forgetting -- read alongside section 9
5. Ouyang et al. (2022). Training language models to follow instructions with human feedback — InstructGPT: the origin of the whole SFT -> RM -> PPO pipeline
6. Wang et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions — generating instruction data from the model itself
7. Zhou et al. (2023). LIMA: Less Is More for Alignment — a few thousand high-quality examples suffice -- why we use only 4,000

The writing, code and notebooks in this series are licensed under CC BY-NC-SA 4.0 — reuse and adapt them freely with attribution, for non-commercial purposes, and share your adaptations under the same terms. Third-party models and datasets referenced here keep their own licences.