Minimum description length

Minimum description length (MDL) is a model selection principle in which the shortest description of the data is judged to be the best model. MDL methods learn through a data compression perspective and are sometimes described as mathematical applications of Occam's razor. The principle has its origins mostly in information theory and has been further developed within statistics, theoretical computer science and machine learning, and more narrowly computational learning theory. The phrase is used in different but interrelated ways. In Jorma Rissanen's theory of learning, models are statistical hypotheses and descriptions are defined as universal codes, and in 1978 Rissanen published an MDL learning algorithm that used the statistical notion of information rather than algorithmic information. Within algorithmic information theory the description length of a data sequence is the length of the smallest program that outputs it, an idealized form of MDL that is closely related to Solomonoff's theory of inductive inference.