Gibbs Sampling Algorithm

Gibbs sampling is a Markov chain Monte Carlo algorithm for drawing samples from a complicated joint probability distribution in cases where sampling from that joint distribution directly is difficult but sampling from each variable's own conditional distribution, given the current values of every other variable, is practical. It works by cycling through the variables one at a time, replacing each one's value with a fresh draw from its conditional distribution, and repeating this process until the resulting chain of samples settles into a good approximation of the true target distribution. The method is named after the physicist Josiah Willard Gibbs, not because he devised it but because its logic resembles an analogy drawn from statistical physics. The brothers Stuart and Donald Geman gave the algorithm its formal description in 1984, and it went on to become a standard tool in Bayesian statistics for computing posterior and marginal probability distributions that would otherwise be very hard to calculate directly.