Information bottleneck method

The information bottleneck method is a technique from information theory for finding the best tradeoff between accuracy and compression when summarizing a random variable, for example by clustering it, while keeping as much information as possible about a second, relevant variable. It was introduced by Naftali Tishby, Fernando C. Pereira and William Bialek. The method generalizes the classical idea of a minimal sufficient statistic from parametric statistics to arbitrary distributions by relaxing the sufficiency condition, so that a compressed representation only has to keep some fraction of the mutual information with the relevant variable. It can also be read as a rate distortion problem, which leads to a general iterative algorithm for solving the tradeoff and for computing the information curve. Applications include distributional clustering and dimension reduction, and it has more recently been proposed as a theoretical foundation for deep learning.