Knowledge distillation

Knowledge distillation, also called model distillation, is a machine learning technique for transferring knowledge from a large model to a smaller one. Very deep neural networks and ensembles of many models have more knowledge capacity than small models, but that capacity may be poorly used, and evaluating them is just as expensive. Distillation trains a smaller model to reproduce the behavior of the larger one, so that it can be deployed on less powerful hardware such as a mobile device. The key idea is that the pattern of values a large model assigns across its outputs, not only its top answer, carries information about how it represents knowledge, and the small model is trained to match that pattern. The technique should not be confused with model compression, which shrinks a model itself without training a new one. A less common variant, reverse knowledge distillation, moves knowledge from a smaller model to a larger one. Distillation has been applied to object detection, acoustic models, natural language processing and graph neural networks. An early form was described as model compression in 2006.