I very recently read where the Merck V920 Ebola Zaire vaccine received a positive opinion form the Committee for Medicinal Products for Human Use (CHMP) of the European Medicines Agency (EMA), which recommended the vaccine for conditional marketing authorization [1]. This reminded me of an article in the January 4th, 2019 issue of STAT Health where I came across an article on the Ebola outbreak that is currently ongoing in the Democratic Republic of Congo [2]. Besides battling the unseen virus, health care workers from WHO and other governmental and non-governmental agencies are risking their lives in regions where rebels are fighting Congolese forces. This was and is complicating medical relief efforts and very importantly the testing of the experimental V920 Ebola vaccine from Merck.
What caught my attention were some of the statistics and claims that were made in the STAT Health article. Two quotes from the article are given below.
- “The Ebola outbreak in the Democratic Republic of the Congo is the second worst on record, having topped 600 cases. But the case count would be much higher still if an experimental Ebola vaccine were not being used to contain spread of the disease, the director-general of the World Health Organization said Thursday.” [This was attributed to Tedros Adhanom Ghebreyesus, the director-general of WHO.]
- “The assessment of vaccine’s effectiveness by Tedros, as he is known, is based on the fact that the case count hasn’t grown exponentially, not on modeling calculations.”
The latter statement made me chuckle: “not on modeling calculations.” Yet in fact, it is based on their observation that the number of cases is NOT following an exponential model. Clearly, it is based on modeling calculations. The following diagram is strictly illustrative of the point (I do not have access to the data). One cannot claim that lives have been saved without some sort of underlying model of expected deaths in the absence of the vaccine.
While this is what immediately struck me, it got me to thinking about the notion of implicit and explicit models. Most everything I have read on the topic of human cognition declares that our minds (consciously or subconsciously) are always working to categorize events/people and to put them into the context of our lives and experiences. Perhaps our minds tell us (consciously or subconsciously) “The last time I ran into someone dressed like that or behaving like that, they were not very friendly or they were not very open to my ideas.” Then we may act more cautiously or more boldly when we share an idea in a meeting. Our minds are constantly building models of the world so that we can make decisions on what to expect and how to act in that world.
I will use this occasion to make this point in the context of frequentist versus Bayesian statistics. As many of my readers may know, I am not educated as a Bayesian, but rather a more recent convert to this way of thinking (as I learned more about it). As I have espoused any Bayesian analysis or approach, the most persistent resistant questions are, “Where does the prior come from?” or “Isn’t the prior subjective?” Of course this is true, BUT WHO CARES! Upon completion of a frequentist analysis and its resulting p-value, the interpretation of the result is usually put into the context of prior knowledge or belief.
For example, if a researcher from a relatively unknown lab produces a novel experimental drug derived from a common dandelion weed, then does a clinical study in pancreatic cancer and declares that the new drug is effective because he/she obtained a p-value of 0.04 for extending overall survival, would not the whole world be skeptical of the finding? Why? Because we know that pancreatic cancer is very difficult to treat. Because we know that many treatments have failed and even the best treatments (approved by regulators through rigorous clinical development programs) based on the best molecular biological knowledge provide only modest gains in survival. Thus, it is difficult to put any p-value from a single experiment into context without some underlying (mathematical or mental) model of what to expect.
Such models are often implicit (subconscious) and based on the individual scientist’s knowledge and experience. One scientist knows the researcher and has more trust in the results. Another researcher doesn’t believe that anything useful could come from a dandelion plant and disregards the findings. Should the findings be pursued with more research? Maybe or maybe not. It depends on whether one thinks it is throwing good money after bad or whether any potentially positive finding for a difficult and terminal disease is worthy of additional research.
The argument for a Bayesian approach is that at least it attempts to quantify the prior knowledge in explicit mathematical expressions (conscious). While there is subjectivity to this endeavor, at least the assumptions and expectations are transparent. As such, this is one reason (there are others that have been noted in other blogs) I tend to favor the Bayesian paradigm. My consulting experience for the last 38 years in pharma related research has taught me that surfacing implicit knowledge and understanding brings greater clarity to any scientific discussion. Phrases like, “There is a reasonable chance of XYZ …” are refined to “There is a 35% chance of XYZ …” or better yet, “The distribution for the measurement of XYZ is …”
I end with my favorite quote about models (no, it’s not George Box!). It comes from a most unusual place … a geo-political book entitled The Clash of Civilizations and the Remaking of World Order by Samuel P. Huntington. I hope you enjoy this lucid description of cognition, cartography and modeling. Huntington wrote this:
“If we are to think seriously about the world, and act effectively in it, some sort of simplified map of reality, some theory, concept, model, paradigm, is necessary. Without such intellectual constructs, there is, as William James said, only ‘a bloomin’ buzzin’ confusion.’ Intellectual and scientific advance, Thomas Kuhn showed in his classic [book] The Structure of Scientific Revolutions, consists of the displacement of one paradigm, which has become increasingly incapable of explaining new or newly discovered facts, by a new paradigm, which does account for those facts in a more satisfactory fashion. ‘To be accepted as a paradigm,’ Kuhn wrote, ‘a theory must seem better than its competitors, but it need not, and in fact never does, explain all the facts with which it can be confronted.’ ‘Finding one’s way through unfamiliar terrain,’ John Lewis Gaddis also wisely observed, ‘generally requires a map of some sort. Cartography, like cognition itself, is a necessary simplification that allows us to see where we are, and where we may be going.‘ [My emphasis added since it is my favorite sentence in the whole book.]
“Simplified paradigms or maps are indispensable for human thought and action. On the one hand, we may explicitly formulate theories or models and consciously use them to guide our behavior. Alternatively, we may deny the need for such guides and assume that we will act only in terms of specific ‘objective’ facts, dealing with each case ‘on its merits.’ If we assume this, however, we delude ourselves. For in the back of our minds are hidden assumptions, biases, and prejudices that determine how we perceive reality, what facts we look at, and how we judge their importance and merits. We need explicit or implicit models … to be able to:
1. order and generalize about reality;
2. understand causal relationships;
3. anticipate and, if we are lucky, predict future developments;
4. distinguish what is important from what is unimportant; and
5. show us what paths we should take to achieve our goals.
“Every model or map is an abstraction and will be more useful for some purposes than for others. A road map shows us how to drive from A to B, but will not be very useful if we are piloting a plane, in which case we will want a map highlighting airfields, radio beacons, flight paths, and topography. With no map, however, we will be lost. The more detailed a map is the more fully it will reflect reality. An extremely detailed map, however, will not be useful for many purposes. If we wish to get from one big city to another on a major expressway, we do not need and may find confusing a map which includes much information unrelated to automotive transportation and in which the major highways are lost in a complex mass of secondary roads. A map, on the other hand, which had only one expressway on it would eliminate much reality and limit our ability to find alternative routes if the expressway were blocked by a major accident. In short, we need a map that both portrays reality and simplifies reality in a way that best serves our purposes.”
Now that is some really good analytical thinking!
References
[1] Merck Receives EU CHMP Positive Opinion for Investigational V920 Ebola Zaire Vaccine for Protection Against Ebola Virus Disease. https://www.mrknewsroom.com/news-release/ebola/merck-receives-eu-chmp-positive-opinion-investigational-v920-ebola-zaire-vaccine- (accessed 23 Oct 19).
[2] STAT Health (4 Jan 2019). WHO’s Tedros: Experimental Ebola vaccine in the DRC has saved countless lives. https://www.statnews.com/2019/01/04/ebola-vaccine-tedros-drc/?utm_source=STAT+Newsletters&utm_campaign=8cc59af83f-MR_COPY_08&utm_medium=email&utm_term=0_8cab1d7961-8cc59af83f-150963065
analytixthinking.blog 
Steve, one inherent deficiency with a Bayesian framework is that it demands a very specific mathematical form of prior knowledge, designed to be fed into statistical modeling machinery. Much scientific knowledge can be encoded in explicit models, but lacks this specific Bayesian form. Let’s take an extreme example. In Huntington’s case of the Kuhnian paradigms, two competing “incommensurate” paradigms are not able to transfer information to each other in the form of Bayesian priors. The fundamental concepts of the two paradigms have too little in common to even do that. Copernicus and Kepler could not use Ptolemaic epicycles to create priors for heliocentric astronomical parameters….they had to go straight to the actual data. Thankfully they also did not have the notions of “statistical inference” to distract them. Elsewhere I have written in praise of statistical thinking, but it must never be given a monopoly on how we think about models in science, engineering, and medicine. See my related comments on non-probability models at https://magazine.amstat.org/blog/2015/05/01/sciencenotartii/
LikeLike
Chris,
Thank you very much for your comments. Yes, there are some situations in which statistical models and the use of priors may not be relevant or appropriate. While I am not familiar with exactly how Copernicus and Kepler arrived at their mathematical formulations of elliptical orbits of our planets and what data or analysis they did, I do think that the unraveling or discovery of physical laws of nature are a different beast than trying to understand something as complex and diverse as biology or even more so the human psyche or social phenomenon. In those latter scientific areas, we are much more likely to be trying to probabilize what is likely to be true (does this drug at this dose have a positive effect on this outcome in this patient population over this timeframe?) rather than describing a (truly) universal phenomenon like gravitational forces.
In the arenas of biology, psychology and sociology (the latter two being closer to the world that world that Huntington was describing), we understand a lot less about causal mechanisms as there are many, many factors that create uncertainties that can only be quantified by probability models. In those situations, using Bayesian methods with mathematically described priors is not only reasonable but also doable.
Lastly, with regard to your statement, “one inherent deficiency with a Bayesian framework is that it demands a very specific mathematical form of prior knowledge,” I do not fully agree. As pointed out in a couple of my other blogs, a simple point prior for a hypothesis can suffice for estimating a posterior probability using Bayes Factor. Let’s take an example even from the field of physics – cold fusion. Everything we know about physics would indicate that any prior about a hypothesis for a method to achieve a cold fusion reaction would be very, very, very, very (I can’t write ‘very’ enough) low. So, when it was announced in 1989 by Fleischmann and Pons that cold fusion had be achieved on a simple, small tabletop device, naturally the world was skeptical. What the scientific world was doing (and what we all do to some varying levels of consciousness or sophistication) is combining prior belief with the current data from Fleischmann and Pons to get a posterior probability.
So, a prior for the hypothesis that cold fusion could be achieved in a manner as described by Fleischmann and Pons might be 0.00…01 (insert as many 0’s as you like but at least 5 seems justified). I do not know if they quoted p-values in their paper, but they reported excessive heat significantly above what would be expected with “normal” chemical processes – i.e. requiring nuclear processes. Using the formula I have used in my blogs p1 < {1 + [(1-p0)/p0] / BFB }-1 where p0 is the point prior probability, p1 is the posterior probability of the hypothesis and BFB is the Bayes Factor Bound = BFB=1/[-e * p * ln(p)], I suspect we would get a small posterior probability that cold fusion was achieved. Of course, subsequent experiments were not ever able to reproduce the result of Fleischmann and Ponns, consistent with my expected low posterior probability.
So, the “specific mathematical form of prior knowledge” can be quite simple – a point probability. I can only say that this ‘back-of-the-envelope’ calculation served me very well in my pharmaceutical career and still serves me well as I read about new scientific findings in the literature.
LikeLike
And thank you Steve for your thought provoking ideas and excellent discussion. The examples you provide in your TAS paper are certainly plausible; I am just saying there are major events in the history of science that would not fit into that framework. It is often quality of evidence, not quantity (in the narrowly Bayesian sense of accumulating data to swamp the low prior by boosting the BFB) that proves decisive. Examples include the discovery of energy quantization, and later of quasicrystals (the latter with a vanishingly low prior). To move to the domain of biology, we have the germ theory of disease (see in particular John Snow’s work on the 1854 London cholera outbreak), and much later, the discovery that prions are the etiologic agent of TSEs. An array of evidence of different kinds, from observational data, natural experiments, & designed experiments (assessing various “subhypotheses” of the over-arching theory) were in totality found to be compelling. (And like in Kepler’s case, thankfully Snow, Pasteur, and Koch did *not* know about the ideas of statistical inference, and were thus not as confused as today’s scientists.) In summary, science often moves forward by a creative scientific argument developed along many different lines of evidence, rather than following a narrow, highly formalized Bayesian methodology that demands a large quantity of one kind of evidence.
LikeLike