2 posts tagged with "distillation"

Every time a user messages your chatbot, you attach the same system prompt, hundreds of tokens long — every request, for the lifetime of the system, paid again and again with no end. This chapter moves that block of knowledge from the prompt into the model's weights, using a technique called Context Distillation in its on-policy version (OPCD). The most beautiful part: the teacher and the student are the exact same model — the only thing that differs is who gets to see the prompt.

In chapter 6 we distilled a context into the weights of the same model. In this chapter we distill an entire model into a smaller one. The two chapters are a deliberate pair: context distillation changes what the model knows without having to tell it anymore — model distillation changes the size of the model while trying not to change what it can do. And the heart of this chapter cuts hard against intuition: the most valuable thing a teacher can hand its student is not the right answer, but the way the teacher is wrong — and the dial called temperature is what makes that visible.