Search

## Be Credible by being Bayesian

Recently at work we encountered a problem with the "standard" way of computing confidence intervals for a Binomial proportion using the Normal approximation. That is, At small sample sizes and high sample proportions, this is too conservative. For example, with a sample size of 10, the confidence intervals for a sample proportion of 0.8 at 95% confidence level, the confidence interval based on the Normal approximation is [0.55, 1].

We can easily see that this confidence interval is too wide as it includes p = 1.0, but we already saw 2 negative examples, which flies in the face of the evidence that p = 1.0 should be on the table.

There are many alternative confidence interval computation methods available, but one particular approach which I prefer is Bayesian: Use a Beta prior disitribution and use that to obtain a posterior distribution (Beta-Binomial). For example, using the Uniform (Uninformed) prior distribution Beta(1,1), the confidence interval (or "credible interval") is the much more plausible [0.480.94].

One of the nice properties of this approach is the subjective knowledge that is inherently available as input to the model. The hyperparameters alpha and beta can be interpreted as alpha - 1 is the number of successes and beta - 1 is the number of failures. So if we strongly believe a priori that the sample proportion is around 0.9, then we could use the Beta(10, 2) as the prior distribution instead, which gives the credible interval of [0.64, 0.95].