[LLM 10/10] Deployment: You're Not Waiting on Compute, You're Waiting on Weights to Travel

Across nine chapters we've injected knowledge, taught format, arranged preferences, distilled models, kept models from breaking, and measured honestly. But even our best model is still just a checkpoint file that nobody can call. This final chapter puts it into real service, and proves the single sentence that governs every decision in LLM serving: decoding one token at a time is not limited by compute power, it is limited by memory bandwidth — we'll compute the speed ceiling from the GPU's datasheet before writing a single line of code, then measure the real thing against it.

1. The Problem​

You have a model trained and selected by the sweep in chapter 9. The next questions aren't machine learning questions at all:

Most people answer these with "I tried generate and it felt fast," which isn't engineering. And the way most benchmark articles answer them is usually meaningless, for very specific reasons:

• Reporting tok/s without stating batch size — 27 tok/s at batch 1 and 400 tok/s at batch 32 can be the exact same machine
• Averaging prefill speed (reading the prompt) together with decode (producing the answer), when the two hit entirely different bottlenecks
• Reporting post-quantisation speed without reporting quality — this is the original sin of the genre, and we'll come back to it several times in this chapter

This chapter answers every question above with numbers we measure ourselves, on the same free machine we've used all series.

2. What We're Going to Do​

We start from one physical fact and let everything flow out of it.

The plan for this chapter is straightforward, and I think it's the most self-proving experiment in the series:

1. Compute the ceiling on decode speed from the T4's datasheet — without running anything yet
2. Measure reality with a server deliberately written badly first, and see how many times off the ceiling it lands
3. Close the gap step by step — static KV cache, torch.compile, continuous batching written by hand in about 60 lines — measuring again at every step so we know exactly how much each one bought
4. What it costs — quantise to int8 and nf4, then measure speed and quality on KobEval-TH, always together

3. The Equations​

3.1 The memory budget at serving time​

• $P$ = parameter count, $b_w$ = bytes per weight (fp16 = 2)
• $L$ = number of layers, $n_{kv}$ = number of key-value heads, $d_h$ = dimension per head, $b_{kv}$ = bytes per value in the cache
• $s$ = context length, $B$ = number of sequences held at once
• The leading 2 is one set each for K and V, while $M_{\text{act}}$ at inference time is so small it's nearly droppable

Substituting the real values from Qwen3-0.6B's config.json ($L=28$, $n_{kv}=8$, $d_h=128$, fp16):

Beware the trap people fall into most often right here: Qwen3 uses grouped-query attention, so you must use $n_{kv}=8$, not the number of attention heads (16) — get that one value wrong and your answer doubles instantly. And this 112 KiB figure isn't a number floating in an article. It's asserted in the test suite of this site's widget (memoryMath.test.ts) — the series' code and its prose are forced to agree.

Now multiply by the model's full context ceiling ($s = 40{,}960$, per the real max_position_embeddings):

For a single sequence — roughly 3.9 times the entire model's weights (596M parameters × 2 bytes ≈ 1.19 GB). This is the arithmetic reason long context is expensive: the budget isn't eaten by the model, it's eaten by the conversation's memory.

3.2 The most important equation in this chapter — the decode ceiling​

Producing one token requires reading every weight once, plus the KV cache accumulated so far, so

320 GB/s is the T4's GDDR6 bandwidth straight off the datasheet — ~266 tok/s is the theoretical ceiling at batch = 1. No code on earth makes a T4 decode this model single-stream faster than that, because it's a limit of the wiring, not of the software.

Let's check that bandwidth really is the bottleneck: at 266 tok/s the compute load is $2P \approx 1.19$ GFLOP/token, totalling ~0.32 TFLOPS, or about 0.5% of the 65 TFLOPS (fp16) a T4 can do — the chip is 99.5% idle. The notebook will measure the real number (far below the ceiling), and section 8 will explain and close that gap layer by layer.

3.3 Little's Law — sizing the system you must support​

The number of jobs in the system ($L$) equals the arrival rate ($\lambda$) times the mean time per job ($W$) — always true, with no assumptions about the distribution. You can size a system with it immediately: if users arrive at $\lambda = 5$ requests/second and each answer takes $W = 2$ seconds, the system must hold $L = 10$ requests at once — at an average context of 1,024 tokens that's a KV cache of $10 \times 1{,}024 \times 112\ \text{KiB} \approx 1.2$ GB reserved at all times. Equations 3.1 and 3.3 are one equation seen from two angles.

3.4 INT8 symmetric quantisation​

Store weights as 8-bit integers ($x_q \in [-127, 127]$) with one scale factor $s$ per group, and multiply back when you use them. The error per value is at most $s/2$. What you gain is halving the bytes per weight — and since equation 3.2 says time per token scales with bytes read, in theory decode gets 2× faster. In practice the kernel that has to dequantise can eat that profit entirely or worse (especially bitsandbytes' LLM.int8() on a T4) — measure, never guess. And don't forget: bitsandbytes touches weights only. The KV cache is still fp16 at the same 112 KiB/token.

3.5 Prefill vs Decode — two regimes you must never mix​

Define arithmetic intensity $I$ = FLOPs done per byte read, and compare it against the GPU's "ridge point":

• Decode (batch 1): read 2 bytes of weight, do 2 FLOP → $I \approx 1$ — about 200× below the ridge → bandwidth-bound
• Prefill: a prompt of $s$ tokens is processed at once, so each weight is reused $s$ times per read → $I \approx s$ — a prompt beyond ~200 tokens is already compute-bound

These two phases are entirely different worlds: prefill can push thousands of tokens per second, decode gets tens to hundreds. Anyone who averages the two into a single "tok/s" is reporting a number that tells you almost nothing — which is why our notebook always reports TTFT (time to first token — measuring prefill) and ITL (inter-token latency — measuring decode) separately.

4. Seeing the Equations​

Context is what eats the budget, not the model​

Note the square marker at the lower right: a single sequence at the full 40,960-token context uses 4.7 GB of KV — nearly four times the model's own weights. And the batch-16 line hits the 16 GB ceiling from ~8,070 tokens. This is why LLM providers bill by context length.

Batching: one read of the weights, redeemed for many tokens​

Read this graph on both axes: the blue line rises (good) but the orange line rises too (bad). Batching isn't free — it is selling each individual user's latency to buy the system's throughput. Where you should sit on this graph is a business decision, not a technical one.

The queue explodes before the server fills up​

The point to remember: at 80% utilisation p99 has already blown past 11 seconds, even though the server is "still 20% free." Real systems must always leave headroom — anyone who sizes a system to exactly 100% of measured throughput is designing a system whose p99 is infinite.

Play with the serving budget yourself — switch to Serving mode and adjust context and concurrent requests, and watch when you hit the 16 GB ceiling:

5. Setting Up the Environment​

Open Colab and pick Runtime → Change runtime type → T4 GPU (the free tier suffices — for the last time in this series).

torch_dtype=torch.float16      # not bfloat16 — the last time you'll read this line from me
attn_implementation="sdpa"     # not flash_attention_2
# and calling it in advance: vLLM needs dtype="half" — leave it on auto and it finds bf16 in the config and refuses on the spot

cap = torch.cuda.get_device_capability(0)
print("compute capability:", cap)                    # T4 = (7, 5)
print("native bf16:", cap[0] >= 8)                   # T4 -> False
print("torch says   :", torch.cuda.is_bf16_supported())  # T4 -> True (counts emulation!)

Stage 1 — Merge the adapter and export (~2 min)​

All series long we trained with LoRA, which becomes a burden at serving time: every forward pass has to compute an extra $BAx$. The good news is that LoRA merges back into the base weights in closed form: $W' = W + \tfrac{\alpha}{r}BA$ — once served, it costs exactly what the base model costs.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
).cuda()

policy = PeftModel.from_pretrained(base, "qwen3-th-lora-best")   # the sweep winner from chapter 9
merged = policy.merge_and_unload()                               # W' = W + (α/r)·B·A
merged.save_pretrained("qwen3-th-serve")
tok.save_pretrained("qwen3-th-serve")

Don't take it on faith that merging gives you the same model — prove it by comparing logits:

# Thai prompt: "What are the duties of the Provincial Electricity Authority?"
x = tok("การไฟฟ้าส่วนภูมิภาคมีหน้าที่อะไร", return_tensors="pt").to("cuda")
with torch.no_grad():
    d = (policy(**x).logits - merged(**x).logits).abs().max().item()
print(f"max |Δlogit| = {d:.4f}")   # around ~1e-3 — near zero, but not exactly zero

It isn't exactly zero because adding $\tfrac{\alpha}{r}BA$ into $W$ in fp16 involves rounding — if you see the 1e-3 range, that's normal. If you see the 1.0 range, you loaded the wrong adapter or the dtypes don't match.

The price you pay: a ~20 MB adapter file (10.1M parameters at r = 16) becomes ~1.2 GB of full weights, a ~60× increase, in exchange for every serving tool (vLLM included) seeing it as one ordinary model.

6. Preparing the Data​

The "data" in this chapter isn't a training set, it's a workload — and a workload measured the wrong way always yields a p99 that's too pretty to be true.

• 60 Thai prompts from the same pool as KobEval-TH — a mix of short, medium and long, so prefill has the variety of real work
• The TH-KNOW quality set from chapter 9 — reused for every configuration whose speed we measure
• An open-loop load generator: request arrival times drawn from a Poisson process, then fired on schedule whether or not the server is ready

import numpy as np

rng = np.random.default_rng(42)
gaps = rng.exponential(1.0 / LAM, size=N_REQUESTS)   # Poisson process: gaps ~ Exp(λ)
arrivals = np.cumsum(gaps)                           # the firing schedule — followed strictly

Every request records three values: TTFT, mean ITL, and token count — then we summarise as p50/p99 per configuration.

7. The Main Code​

7.1 Stage 2 — A baseline deliberately made bad (~4 min)​

from fastapi import FastAPI
import threading, uvicorn

app = FastAPI()

@app.post("/generate")
def generate(body: dict):
    ids = tok(body["prompt"], return_tensors="pt").to("cuda")
    out = merged.generate(**ids, max_new_tokens=128, do_sample=False)
    return {"text": tok.decode(out[0, ids.input_ids.shape[1]:],
                               skip_special_tokens=True)}

threading.Thread(
    target=uvicorn.run, args=(app,),
    kwargs=dict(host="127.0.0.1", port=8000, log_level="warning"),
    daemon=True,
).start()

This server violates everything this chapter teaches: one request at a time, no batching, the KV cache reallocated every call, a Python loop per token — and that is its job. It's the baseline every improvement gets measured back against. If you don't measure the starting point, "5× faster" is just advertising.

7.2 Stage 3a — Static KV cache + torch.compile​

Ordinary generate grows the KV cache one token at a time, so shapes change constantly and it can't be compiled. Reserve the whole cache up front (static) and shapes stay fixed enough for torch.compile to capture the entire graph into a CUDA graph — eliminating the baseline's single biggest overhead, launching kernels one at a time from Python.

merged.generation_config.cache_implementation = "static"
fast = torch.compile(merged, mode="reduce-overhead")

warm = tok("อุ่นเครื่อง", return_tensors="pt").to("cuda")   # Thai for "warm up"
for _ in range(3):                                   # always warm up before timing
    fast.generate(**warm, max_new_tokens=8)

7.3 Stage 3b — Continuous batching, ~60 lines written by hand​

Equation 3.2 says reading the weights once is the big cost — batching is dividing that one cost across many tokens. But static batching (wait for $B$ to accumulate, start, then wait for the longest one to finish) wastes enormous room, so continuous batching admits new requests into the batch the moment a slot frees up. The heart of it is this loop:

from collections import deque
queue, running, MAX_BATCH = deque(), [], 16

while queue or running:
    while queue and len(running) < MAX_BATCH:
        running.append(Sequence(queue.popleft()))   # admitted mid-flight, without waiting for the old batch

    step(running)          # one forward step for every sequence — 1 read of the weights, B tokens out

    running = [s for s in running if not s.done]    # finished ones leave immediately, returning slots to the queue

The full version (~60 lines, including per-sequence position ids and masks) is in the notebook. It is not vLLM — no paged memory, no prefix cache — but it proves the mechanism with code you can read in one screen, and every percent of its measured improvement can be accounted for, which matters more than sophistication in this lesson.

7.4 Quantisation — int8 and nf4, and the real price​

from transformers import BitsAndBytesConfig

int8 = AutoModelForCausalLM.from_pretrained(
    "qwen3-th-serve",
    quantization_config=BitsAndBytesConfig(load_in_8bit=True),
    device_map={"": 0},
)

nf4 = AutoModelForCausalLM.from_pretrained(
    "qwen3-th-serve",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16,
    ),
    device_map={"": 0},
)

And a prediction stated plainly in advance: on a T4, bitsandbytes' int8 is often slower than fp16 — LLM.int8() splits outliers off to be multiplied in fp16, so you pay overhead twice. It is a VRAM-saving tool, not a speed tool. If your measurements come out that way, that isn't a bug on your side, it's the truth review articles rarely print.

7.5 Stage 4 — vLLM (optional, and the most fragile thing in the series)​

try:
    from vllm import LLM, SamplingParams          # version pinned in the notebook's install cell
    llm = LLM(
        model="qwen3-th-serve",
        dtype="half",                 # the T4 has no bf16 — must be stated explicitly
        enforce_eager=True,           # skip the CUDA graph that misbehaves on sm_75
        gpu_memory_utilization=0.85,
        max_model_len=4096,
    )
    VLLM_OK = True
except Exception as e:
    VLLM_OK = False
    print("vLLM unavailable on this runtime — safe to skip:", e)

This structure is deliberate: if vLLM fails to install or crashes at init, everything in stages 1–3 is still intact. This chapter's main conclusions don't depend on vLLM at all — it's merely evidence of what paged KV plus well-fused kernels can add compared to our 60-line scheduler.

8. Results​

The notebook writes every number to results.json — the frame is theory from the datasheet against measured reality:

What to expect and how to read it: the naive number will land roughly 10× below the ceiling — don't panic, and don't blame the T4. That gap has traceable, layered origins: a Python loop per token, dozens of kernel launches per step, dynamic KV allocation, syncing back to the CPU during sampling — stage 3 removes these layers one at a time and measures again every time. The total creeping closer and closer to the ceiling (but never touching it, because the ceiling excludes KV, activations and the remaining overhead) is the empirical evidence that equation 3.2 really does describe the actual machine — which is why I call it the series' most solid experiment.

Under load, the second table captures the shape of figure 10.3:

And the eyeball quality check no number can substitute for — the same prompt, answered in fp16 versus nf4:

9. Comparison​

The summary table for the whole chapter — every bit of speed gained has to show what it cost, on the same row:

* At a context of 1,024 tokens, the KV budget from equation 3.1 supports hundreds of sequences (~14.8 GB ÷ 112 MiB ≈ 125) — the real limiter is the scheduler and prefill compute, not VRAM, which is a lesson in itself.

The pattern you should expect:

• compile helps batch 1 the most (it kills per-step overhead, which is the single-stream bottleneck)
• batching barely helps tok/s per request but multiplies the aggregate — and makes p99 worse under high load
• int8 lowers VRAM but is slower on a T4; nf4 lowers VRAM more and is faster than int8 — for quality you must look at the rightmost column yourself, never conclude it from the speed columns
• vLLM, if it survives, should clearly beat our hand-written scheduler at high concurrency — if it doesn't, enforce_eager is eating its profit

Traps to watch for​

1. The server vanishing silently because Colab dropped the connection — measure in short bursts, write results immediately (section 7.1). Don't plan a measurement longer than the session's lifetime.

2. bf16 on a T4 — by chapter ten you should be able to predict this before the error appears: torch_dtype=torch.float16 in transformers and dtype="half" in vLLM. The series' running joke ends with this chapter, but Turing cards live on in the world.

3. No warmup before measuring — the first compile takes minutes; if it leaks into your timing, your conclusion inverts instantly (section 7.2)

4. Reporting tok/s without stating batch size — the same number can mean a system where users wait 40 ms or 500 ms per token. Every number in this chapter therefore carries its @B with it.

5. Mixing prefill into decode — long prompts make "average tok/s" deceptively high, because prefill is compute-bound and consumes thousands of tokens per second (section 3.5) — report TTFT and ITL separately, and only separately.

10. Summary​

• Decode at batch 1 is waiting on weights to travel, not on compute — the chip is ~99.5% idle while "working flat out"
• The speed ceiling is computable from the datasheet: 320 GB/s ÷ 1.2 GB ≈ 266 tok/s, before running a single line of code
• KV cache is 112 KiB/token (the same number this site's test suite asserts) — at the full 40,960-token context that's ~4.7 GB for one single sequence, nearly four times the model's weights — context is what eats the budget
• Batching = selling latency to buy throughput, and continuous batching is the way of selling that loses the least
• Quantisation reduces the bytes that must travel — twice as fast in theory, but in practice you must measure, and always measure quality alongside
• Little's law ties it all together: $L = \lambda W$ gives the number of sequences you must hold, which loops right back into the KV budget
• p99 explodes before the server fills up — a system with no headroom is a system designed to fail exactly when the most people are using it

Closing the series​

The ten chapters built a single path: inject knowledge (ch. 1) → teach format (ch. 2) → arrange preferences three ways (ch. 3–5) → distil it smaller (ch. 6–7) → keep it from breaking (ch. 8) → measure honestly (ch. 9) → put it into service and measure against the ceiling physics sets (this one). But what I really want you to carry away isn't any one technique. It's the habits every chapter repeated: write the equation before writing the code, measure everything with a confidence interval, and always publish your own limitations in the yellow box at the end. Models will change, libraries will change, cards will get ten times faster — those three habits will still hold on the day everything in this series is obsolete.

All ten notebooks run to completion on free Colab — don't take my word for it, go run them and see where your numbers differ from mine. If you wandered in and this is the first chapter you've read: start at chapter 1 — Continue Pretraining and walk the path all the way to here. See you around.

References​

1. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention — PagedAttention: the KV-cache management behind vLLM
2. Yu et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI '22) — the original continuous batching, hand-rolled in miniature in section 7
3. Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — FlashAttention -- and why a T4 cannot use it
4. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — GPTQ: accurate post-training quantization
5. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — LLM.int8(): the basis of bitsandbytes' 8-bit mode
6. Pope et al. (2022). Efficiently Scaling Transformer Inference — a systems-level analysis of inference bottlenecks
7. Williams et al. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures — the roofline model behind section 3's 266 tok/s ceiling
8. Pipatanakul et al. (2023). Typhoon: Thai Large Language Models — Typhoon: another line of Thai LLMs
9. Nguyen et al. (2023). SeaLLMs -- Large Language Models for Southeast Asia — SeaLLMs: models for Southeast Asian languages
10. Pairatsuppawat et al. (2025). SiamGPT: Quality-First Fine-Tuning for Stable Thai Text Generation — SiamGPT: quality-first Thai fine-tuning

The writing, code and notebooks in this series are licensed under CC BY-NC-SA 4.0 — reuse and adapt them freely with attribution, for non-commercial purposes, and share your adaptations under the same terms. Third-party models and datasets referenced here keep their own licences.