protologue

Knowledge Distillation

Also called distillation, model distillation, teacher-student training.

Knowledge distillation is training a smaller student model to imitate a larger teacher model, using the teacher's output probabilities or generated responses as training targets, so that the student keeps much of the teacher's ability at lower cost.

Description

The original method trains on the teacher's full probability distribution, whose soft targets carry more information than single correct labels. With LLMs, distillation is commonly done by fine-tuning a small model on responses, including reasoning traces, generated by a large one, which is how several small reasoning models have been built. Some model providers' terms restrict using their outputs to train competing models.

Sources

  1. Hinton et al. (2015). Distilling the Knowledge in a Neural Network.
  2. DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.

Cite this entry

Protologue. (2026). Knowledge Distillation. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0183). https://protologue.com/t/knowledge-distillation/

BibTeX
@misc{protologue_knowledge_distillation,
  title = {Knowledge Distillation},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0183},
  url = {https://protologue.com/t/knowledge-distillation/}
}

Markdown JSON