# Knowledge Distillation

> Knowledge distillation is training a smaller student model to imitate a larger teacher model, using the teacher's output probabilities or generated responses as training targets, so that the student keeps much of the teacher's ability at lower cost.

- Identifier: PTL-0183
- Category: Foundations
- Canonical URL: https://protologue.com/t/knowledge-distillation/
- Also known as: distillation, model distillation, teacher-student training
- Introduced: 2015

## Description

The original method trains on the teacher's full probability distribution, whose soft targets carry more information than single correct labels. With LLMs, distillation is commonly done by fine-tuning a small model on responses, including reasoning traces, generated by a large one, which is how several small reasoning models have been built. Some model providers' terms restrict using their outputs to train competing models.

## Broader terms

- [Fine-Tuning](https://protologue.com/t/fine-tuning/)

## Related terms

- [Reasoning Model](https://protologue.com/t/reasoning-model/)
- [Model Cascade](https://protologue.com/t/model-cascade/)

## Sources

- Hinton et al. (2015). Distilling the Knowledge in a Neural Network. https://arxiv.org/abs/1503.02531
- DeepSeek-AI et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948

## Cite this entry

Protologue. (2026). Knowledge Distillation. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0183). https://protologue.com/t/knowledge-distillation/

License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)