The world’s leading publication for data science, AI, and ML professionals.

Making sense of conjugate priors

Learn how, in some cases, you can calculate posterior distributions by

hand!

Photo by Scott Graham on Unsplash
Photo by Scott Graham on Unsplash

Suppose you’ve tossed a coin 1,000 times and obtained 292. You’d like to know what the Probability of obtaining heads is from a single coin toss – but you don’t just want a single estimate, you want an entire distribution. If you define

  • y: the number of heads you obtain
  • θ: the probability of obtaining heads from a single coin toss

and then model y as a binomial distribution with n=1,000, then the posterior distribution is very easy to obtain with just a few lines of code:

Image by author
Image by author

But…could we have calculated this distribution analytically without PyMC3’s help?

Bayes’ theorem

From Bayes’ theorem, we have:

Image by author
Image by author

Applying this to our problem, we obtain

Image by author
Image by author

Substituting the definitions of the two distributions, we can re-write the right-hand-side as

Image by author
Image by author

Removing constants (that don’t depend on theta), we can substitute this into the equation above to obtain

Image by author
Image by author

which we can recognise (by inspection) as a Beta(292+1, 1000–292+1) distribution. Because our posterior and our prior are from the same family, the Beta distribution is known as a _conjugate prior_ for the Binomial distribution.

If we try plotting this using scipy, we’ll get the same distribution that PyMC3 has calculated for us:

Image by author
Image by author

That’s the nice thing about conjugate priors: they make posterior probabilities analytically tractable.

Calculating posteriors by hand is an excellent exercise in deepening your understanding. However, in general, we will not find ourselves in such fortunate situations – that’s when we need tools such as PyMC3.

Further reading

For an excellent and practical guide to Bayesian statistics, I recommend reading Bayesian Analysis with Python and listening to the Learning Bayesian Statistics podcast.


Related Articles