[LLM 10/10] Deployment: You're Not Waiting on Compute, You're Waiting on Weights to Travel

Across nine chapters we've injected knowledge, taught format, arranged preferences, distilled models, kept models from breaking, and measured honestly. But even our best model is still just a checkpoint file that nobody can call. This final chapter puts it into real service, and proves the single sentence that governs every decision in LLM serving: decoding one token at a time is not limited by compute power, it is limited by memory bandwidth — we'll compute the speed ceiling from the GPU's datasheet before writing a single line of code, then measure the real thing against it.

Open in Colab10_deployment.ipynb

Thai LLM seriesPart 10 of 10

1. 1Continue Pretraining
2. 2SFT and LoRA
3. 3RLHF and PPO
4. 4DPO: Direct Preference Optimization
5. 5GRPO
6. 6Context Distillation
7. 7Model Distillation
8. 8Guardrails
9. 9Benchmarking
10. 10Deployment(you are here)

1. The Problem

You have a model trained and selected by the sweep in chapter 9. The next questions aren't machine learning questions at all:

The question from the person paying	The number that answers it
How long does one user wait	p50 / p99 latency
How many concurrent users can we take	concurrency (Little's law)
How many GPUs do we need	throughput (tok/s)
How long a context can we allow	KV cache budget

Most people answer these with "I tried generate and it felt fast," which isn't engineering. And the way most benchmark articles answer them is usually meaningless, for very specific reasons:

• Reporting tok/s without stating batch size — 27 tok/s at batch 1 and 400 tok/s at batch 32 can be the exact same machine
• Averaging prefill speed (reading the prompt) together with decode (producing the answer), when the two hit entirely different bottlenecks
• Reporting post-quantisation speed without reporting quality — this is the original sin of the genre, and we'll come back to it several times in this chapter

This chapter answers every question above with numbers we measure ourselves, on the same free machine we've used all series.

2. What We're Going to Do

We start from one physical fact and let everything flow out of it.

The core idea of this chapter

When decoding one token at a time at batch = 1, producing a single token requires reading every weight of the model out of HBM, one full pass, while doing only ~2 FLOP of compute per weight — so the GPU sits idle waiting for data to arrive. You aren't waiting on compute, you're waiting on weights to travel from memory to the chip.

Every serving optimisation that means anything — batching, quantisation, paged KV cache — attacks this same bottleneck from a different angle: reduce the bytes that must travel, or get more out of each trip.

The plan for this chapter is straightforward, and I think it's the most self-proving experiment in the series:

1. Compute the ceiling on decode speed from the T4's datasheet — without running anything yet
2. Measure reality with a server deliberately written badly first, and see how many times off the ceiling it lands
3. Close the gap step by step — static KV cache, torch.compile, continuous batching written by hand in about 60 lines — measuring again at every step so we know exactly how much each one bought
4. What it costs — quantise to int8 and nf4, then measure speed and quality on KobEval-TH, always together

3. The Equations

3.1 The memory budget at serving time

$$M \;=\; \underbrace{P\,b_w}_{\text{weights}} \;+\; \underbrace{2\,L\,n_{kv}\,d_h\,s\,B\,b_{kv}}_{\text{KV cache}} \;+\; M_{\text{act}}$$

• $P$ = parameter count, $b_w$ = bytes per weight (fp16 = 2)
• $L$ = number of layers, $n_{kv}$ = number of key-value heads, $d_h$ = dimension per head, $b_{kv}$ = bytes per value in the cache
• $s$ = context length, $B$ = number of sequences held at once
• The leading 2 is one set each for K and V, while $M_{\text{act}}$ at inference time is so small it's nearly droppable

Substituting the real values from Qwen3-0.6B's config.json ($L=28$, $n_{kv}=8$, $d_h=128$, fp16):

$$\text{KV/token} \;=\; 2 \times 28 \times 8 \times 128 \times 2 \;=\; 114{,}688 \text{ bytes} \;=\; 112\ \text{KiB exactly}$$

Beware the trap people fall into most often right here: Qwen3 uses grouped-query attention, so you must use $n_{kv}=8$, not the number of attention heads (16) — get that one value wrong and your answer doubles instantly. And this 112 KiB figure isn't a number floating in an article. It's asserted in the test suite of this site's widget (memoryMath.test.ts) — the series' code and its prose are forced to agree.

Now multiply by the model's full context ceiling ($s = 40{,}960$, per the real max_position_embeddings):

$$114{,}688 \times 40{,}960 \;=\; 4{,}697{,}620{,}480 \text{ bytes} \;\approx\; 4.7\ \text{GB} \;(\approx 4.4\ \text{GiB})$$

For a single sequence — roughly 3.9 times the entire model's weights (596M parameters × 2 bytes ≈ 1.19 GB). This is the arithmetic reason long context is expensive: the budget isn't eaten by the model, it's eaten by the conversation's memory.

3.2 The most important equation in this chapter — the decode ceiling

Producing one token requires reading every weight once, plus the KV cache accumulated so far, so

$$t_{\text{token}} \;\gtrsim\; \frac{M_{\text{weights}} + M_{\text{KV}}}{\text{BW}} \qquad\Longrightarrow\qquad \text{tok/s} \;\lesssim\; \frac{320\ \text{GB/s}}{1.2\ \text{GB}} \;\approx\; 266$$

320 GB/s is the T4's GDDR6 bandwidth straight off the datasheet — ~266 tok/s is the theoretical ceiling at batch = 1. No code on earth makes a T4 decode this model single-stream faster than that, because it's a limit of the wiring, not of the software.

Let's check that bandwidth really is the bottleneck: at 266 tok/s the compute load is $2P \approx 1.19$ GFLOP/token, totalling ~0.32 TFLOPS, or about 0.5% of the 65 TFLOPS (fp16) a T4 can do — the chip is 99.5% idle. The notebook will measure the real number (far below the ceiling), and section 8 will explain and close that gap layer by layer.

3.3 Little's Law — sizing the system you must support

$$L \;=\; \lambda\,W$$

The number of jobs in the system ($L$) equals the arrival rate ($\lambda$) times the mean time per job ($W$) — always true, with no assumptions about the distribution. You can size a system with it immediately: if users arrive at $\lambda = 5$ requests/second and each answer takes $W = 2$ seconds, the system must hold $L = 10$ requests at once — at an average context of 1,024 tokens that's a KV cache of $10 \times 1{,}024 \times 112\ \text{KiB} \approx 1.2$ GB reserved at all times. Equations 3.1 and 3.3 are one equation seen from two angles.

3.4 INT8 symmetric quantisation

$$s \;=\; \frac{\max|x|}{127},\qquad x_q \;=\; \mathrm{round}\!\left(\frac{x}{s}\right),\qquad \hat{x} \;=\; s\,x_q$$

Store weights as 8-bit integers ($x_q \in [-127, 127]$) with one scale factor $s$ per group, and multiply back when you use them. The error per value is at most $s/2$. What you gain is halving the bytes per weight — and since equation 3.2 says time per token scales with bytes read, in theory decode gets 2× faster. In practice the kernel that has to dequantise can eat that profit entirely or worse (especially bitsandbytes' LLM.int8() on a T4) — measure, never guess. And don't forget: bitsandbytes touches weights only. The KV cache is still fp16 at the same 112 KiB/token.

3.5 Prefill vs Decode — two regimes you must never mix

Define arithmetic intensity $I$ = FLOPs done per byte read, and compare it against the GPU's "ridge point":

$$I_{\text{ridge}} \;=\; \frac{65\ \text{TFLOPS}}{320\ \text{GB/s}} \;\approx\; 203\ \text{FLOP/byte}$$

• Decode (batch 1): read 2 bytes of weight, do 2 FLOP → $I \approx 1$ — about 200× below the ridge → bandwidth-bound
• Prefill: a prompt of $s$ tokens is processed at once, so each weight is reused $s$ times per read → $I \approx s$ — a prompt beyond ~200 tokens is already compute-bound

These two phases are entirely different worlds: prefill can push thousands of tokens per second, decode gets tens to hundreds. Anyone who averages the two into a single "tok/s" is reporting a number that tells you almost nothing — which is why our notebook always reports TTFT (time to first token — measuring prefill) and ITL (inter-token latency — measuring decode) separately.

The full lesson is in the course

This post is roughly the first 30% of the chapter. The rest — environment setup, data preparation, the main code, measured results and the wrap-up — is in the free LLM Finetuning course. Sign in with Google to read it.

Read the full lesson in the course →