Improving statistical models using real-world predictions from experts

Medical research often relies on combining hard data with expert intuition, such as a doctor's clinical intuition. To merge these two types of information, statisticians often use Bayesian models, which capture expert knowledge as mathematical probabilities.

This can be difficult to do in practice, because while a doctor might have an excellent intuition about patient outcomes – such as estimating that 8 out of 10 patients will respond to a treatment – Bayesian methods require them to translate that gut feeling into the abstract mathematical notion of uncertainty called a "prior distribution". This can be difficult to do. Indeed, even statisticians who understand the mathematics of prior distributions can find this hard to do for complex biomedical models.

New research led by Robert Goudie and Andrew Manderson, published in the Journal of the American Statistical Association, aims to address this problem. Rather than making experts think about abstract mathematical probabilities, our method allows them to describe the real-world outcomes they expect to see. Our new algorithm then takes these expected real-world outcomes (the "predictive distribution") and mathematically reverse-engineers the "prior distribution" that best matches these outcomes.

A strength of our approach is that it is highly adaptable, working for everything from simple coin flips to complex biomedical models. We demonstrate our method through several realistic examples, including in models of survival rates for curable diseases, which are used in disease areas such as cancer and eye diseases. 

Senior author and Group Leader at the BSU, Robert Goudie explains:

"In many medical settings, if we do not use both observed data and knowledge gleaned from medical experts, our findings may not reflect the complexity of real-world medicine. However, combining both is often challenging. Our work shows how this process can be simplified by asking experts to make predictions about future observations, which we then translate into the mathematical prior distributions needed for Bayesian analysis."