Witten-Bell discounting

Witten-Bell discounting is a smoothing technique used in natural language processing to address the sparse data and zero frequency problems that arise in N-gram language models. It was developed by Ian Witten and Timothy Bell and published in 1991. An N-gram model estimates the probability of a word sequence from how often that sequence was observed in training text, but many legitimate sequences never appear in any finite training set and would otherwise be assigned zero probability, which badly damages a model's performance on new text. Witten-Bell discounting addresses this by using the number of word types that were each observed exactly once in the training data to estimate how much probability mass should be set aside for sequences never seen at all, then redistributing that held out mass across the unseen possibilities. The method combines estimates from different N-gram orders through linear interpolation, weighting each order's contribution with parameters derived from the observed counts, so that a model backs off gracefully from specific, data sparse higher order N-grams to more reliable lower order ones instead of assigning outright zero probability to a novel sequence.