I ran into the topic of Gibbs sampling recently and realized I hadn’t used the technique in a long time. So I figured I’d post a brief explanation of exactly what Gibbs sampling is. I quickly remembered that, contrary to the impression you might get by browsing the Internet for information about Gibbs, Gibbs sampling is extremely complex and even a basic description would require many pages.
So, instead, I just did a quick demo. Gibbs sampling is a technique (MCMC — a huge topic) to approximate drawing samples from a multivariate joint probability distribution (huge topic) when you know the individual conditional distributions (huge topic).
You can see right away that even explaining the type of problem that Gibbs sampling solves requires a fantastic amount of background so I won’t try.
Here’s a concrete example. Suppose you want to draw samples from a bivariate Gaussian distribution (huge topic) with means and covariance (huge topic):
u1 = 0.0, u2 = 0.0 Covar = 1.0 0.6 0.6 1.0
As it turns out, the true graph (huge topic) of this joint bivariate probability distribution looks like (from Wikipedia):
Gibbs sampling can be used to approximate this distribution (gigantic topic), if you know the individual conditional distributions, which are (from: theclevermachine.wordpress.com/2012/11/05/mcmc-the-gibbs-sampler/):
I wrote a Python program (big topic) to approximate the true distribution, dropped the results into Excel, and made a graph. You can see the approximation is close to the true distribution.
As I wrote this blog post, I was stunned by how many complex, interrelated ideas there were. I guess the moral of the story is, that if you want to learn what Gibbs sampling is, buckle up because you’ll have to spend many days just learning the background information, and then many more days learning about Gibbs.