I am writing this Blog because I just got done watching a segment on 60 Minutes about the use of fluvoxamine as a treatment for mild COVID-19 patients. (In case there are non-US folks reading this, 60 Minutes is quite a famous and long-running TV news show that highlights 3 stories for ~20 minutes each Sunday evening that is aired across the US.) That story I am referring to was based on an article in the Journal of the American Medical Association (JAMA) about a randomized, double-blind, placebo-controlled study of fluvoxamine that showed favorable results in patients with milder symptoms of the COVID-19 disease who were treated within 7 days of the onset of symptoms. The study team assessed clinical deterioration as defined by “meeting both criteria of (1) shortness of breath or hospitalization for shortness of breath or pneumonia and (2) oxygen saturation less than 92% on room air or need for supplemental oxygen to achieve oxygen saturation of 92%or greater.” Based on that criteria, 0 of 80 patients on fluvoxamine exhibited clinical deterioration while 6 of 72 placebo patients exhibited clinical deterioration [95 % confidence interval of (1.8%, 16.4%) with a significant p-value = 0.009]. [1]
The 60 Minutes segment conveyed a fair amount of enthusiasm for the results of this trial and therefore, treatment with fluvoxamine. To be fair, the primary investigator, Dr. Eric Lenze, was cautiously optimistic as was Dr. Francis Collins, Director of NIH.
Given the plethora of claims about beneficial treatments that have emerged almost incessantly for the treatment of COVID-19 over the last year, I thought it would be worthwhile to examine this in a little more detail. On the positive side, I was impressed with the high standard of the trial – randomized, double-blind, placebo-controlled – as were the Editors of JAMA. [2] On the negative side, fluvoxamine (Luvox®) is an SSRI for the treatment of depression, social anxiety disorder and obsessive-compulsive disorder, and I am quite skeptical about the “re-purposing” of old treatments to treat COVID-19, especially ones that appear to be far outside the realm of treating an infectious or viral disease.
I will get right to the point: I am going to take a Bayesian approach to this problem as I have done in a number of other Blogs, using my favorite framework for assessing evidence.
What do the data say?
What do I believe?
What do I decide?
What do the data say?
The study is well designed and conducted (the sample size of 152 total patients was based on very reasonable assumptions and power calculations), and the analysis of time-to-clinical worsening (survival analysis by logrank test ) produced a p-value of 0.009. That’s positive … sort of. There are lots of caveats that should be mentioned when assessing the strength of this p-value. To their credit, the authors identified many of these potential shortcomings in their paper as well as mentioning on the 60 Minutes show that a larger, longer trial was needed to confirm this initial result.
However, such caveats in the context of a news show sometimes get lost as a sideline comment as the promise/hope/hype of the results are more fun to talk about than the weaknesses of the research. The principal investigator for the study, Dr. Lenze, said in his initial response to the interviewer, “So, the results were really pretty incredible.” [3] I bet that’s what sticks with most people.
Caveats
The study was done completely remotely, meaning that the patients never actually saw a physician. The assessment of clinical worsening was done by the patients in their homes with equipment (oximeters and self-administered rating scales/questionnaires) provided by the research team. At best, the study team followed up with patients by phone interview.
The study used a 15-day follow-up for the primary analysis. Other secondary measures of efficacy were mixed in terms of benefit or had too few events to be evaluated statistically. Most notably, at 30-days post randomization, both treatments had only one 1 post-trial event of” emergency department visit or hospitalization.”
The study was conducted in a single metropolitan area – St. Louis, Missouri by a Washington University research team. The possibility of an overly homogenous group of patients must be considered.
The study of 152 patients for 15 days is quite small for making a big decision about the adequacy of a new, and unproven treatment that is not presently known for treating an infectious disease.
As for follow-up and completeness, 18/80 (22.5%) fluvoxamine and 19/72 (26.4%) placebo patients did not have a 15-day follow-up assessment. With a small sample size, having about one-quarter of the patients with no protocol defined 15-day evaluation is a bit disturbing. The authors make some arguments about the missing data not being a problem, but I would need to learn more about this and dig into the actual data to be convinced. Everything may be OK here, but I am always nervous about what might be lurking in the patients who failed to complete the study as planned.
So, the p-value of 0.009 is starting off on a bit of shaky ground.
What do I believe?
This is where I take a Bayesian approach, which requires the formulation of a prior belief in the null hypothesis of no difference in fluvoxamine and placebo with respect to the primary outcome measure of clinical deterioration, i.e., fluvoxamine does not work. Alternatively, one can take the complementary probability that fluvoxamine does work, i.e., that the alternative hypothesis is true. I like taking the latter approach because most scientific endeavors are meant to demonstrate that the alternative hypothesis is true, and I like to express the probability of that state of Nature. Furthermore, my experience has shown that in talking to scientists or clinicians, they are more comfortable stating their belief the new or experimental treatment works.
So, where to start?
The JAMA article cites some preclinical research that was published. The article states, “Previous studies have shown that fluvoxamine, a selective serotonin reuptake inhibitor (SSRI) … reduced damaging aspects of the inflammatory response during sepsis, … and decreased shock in murine [mouse] sepsis models.” [4] This is supported by other laboratory experiments using isolated cell lines and investigating the biochemical pathways and mechanisms by which fluvoxamine acts in tissues. [5] As discussed in the 60 Minutes newscast, it was the mouse study of sepsis that caught the eye of Dr. Angela Reierson, also at Washington University, that started the communication with Dr. Lenze and the whole process of pursuing fluvoxamine for treating COVID-19.
In forming my prior, I think like this.
1 – The initial experiments were done in rat brain membranes in a lab.
2 – The sepsis study was done in mice. It included a series of laboratory experiments evaluating the survival of mice that were infected so as to produce sepsis. Each experiment included approximately a dozen mice on control or fluvoxamine as well as some other treatments.
Note 1: As I heard one researcher state in a presentation one time, “We have dozens of treatments that cure stroke in rats, but none in man.” Rats and mice are not humans, and labs are not medical clinics. The animals used are generally bred to be very homogeneous (unlike humans) and the laboratory conditions are exceedingly well-controlled (unlike the life humans live).
Note 2: Again, I am no biologist or clinician, but COVID-19 is not sepsis. They both have an inflammatory component, but they are different diseases, and success in one does not necessarily mean that success will follow in another. Perhaps a study in rats or mice induced with COVID-19 would have been an interesting follow-up study to do.
Note 3: Anyone who has spent more than 1 year working at a pharmaceutical company learns that 10’s of thousands of drugs perform well in the lab and fail to show any benefit in human clinical trials.
From this information alone, my prior could be quite low for the hypothesis that fluvoxamine works. And rather than be vague with terms like “long shot” or “minimal,” I will quantify it as 1/1000, even though I think 1/10,000 is not out of the logical range of possible priors.
But there is more.
Many treatments are being thrown at the COVID-19 pandemic with some biological rationale. Somebody read a paper somewhere or saw that treatment X that was used for disease Y had some effect that might be useful for COVID-19. No one with any ethical standards or sincere scientific interest would give a sick COVID-19 patient a treatment unless they didn’t have some reason to believe that it might work. Many researchers and physicians are trying many different therapeutic approaches, all with some basis in science for why they are doing so. So, fluvoxamine is not the first or only off-label treatment with some biological rationale that has been hurled at this pandemic. Unfortunately, many of these are small and poorly designed studies with no control groups, no randomization, no blinding, etc. and may be hindering our societal search for effective treatments. [6] Little has emerged in any credible way from the hundreds (maybe thousands) of such studies.
Note 4: This is why Dr. Lenze and colleagues are to be lauded for their deliberate and scientific approach to addressing fluvoxamine efficacy in a well-designed study, albeit with the many caveats noted in the previous section.
So, yet another treatment being used in an unusual setting, even with a purported biological basis for its potential therapeutic benefit does not move me much for increasing my prior belief that fluvoxamine works.
OK, as I have said many times in this Blog, I am no biologist or clinician. I only play one in this Blog! So let’s turn to the experts and see what they have to say.
As reported in the 60 Minutes interviews, it seems that the prior of experts was also quite low. Let’s look at what was said. When interviewing the venture capitalist, Steve Kirsch, who threw some money into the endeavor to support the clinical trial, the interview went like this.
- Sharyn Alfonsi: Tell me about the first conversation you had with Dr. Lenze.
- Steve Kirsch: You know, we were like, oh, we– we got a grant application. This is thrilling to us. And it’s for $67,000… and so it’s a very modest amount, so we ran it through the scientific advisory board, and they said, you know, this is novel.
I am always a bit more nervous when experts describe a therapeutic approach as novel. That means untested, unproven, outside the bounds of conventional thinking, etc. Not that this is a bad thing, but it does mean my prior remains skeptical.
And what did the principal investigator, Dr. Lenze, think about the whole endeavor at the beginning. Let’s see what he had to say in the interview.
- Sharyn Alfonsi: Your colleague had to read the study. Silicon Valley guy had to step in. Then there’s some people at a race track that are gonna try it out. It seems unbelievable.
- Dr. Eric Lenze: If you had told me what the odds were at the start of this, I might have reconsidered doing this.
- Sharyn Alfonsi: It was a long shot.
- Dr. Eric Lenze: For sure.
There you have it! Dr. Lenze thought it was a long shot. Unbelievable. Translated … an extremely low prior.
So, my initial assessment of 1/1000 may be in the ballpark. It is hard to know when people use un-quantitative terms such as “long shot” and “unbelievable.” That’s why we need to think seriously about our hypotheses and put meaningful and credible estimates of prior probability on them. For Dr. Lenze, a “long shot” might mean 1/10 or 1/20. To someone else, the “long shot” is 1/50 or 1/100 or even 1/1000. It’s hard to say what is in a person’s mind when they use such a vague phrase.
I will be generous (at least in my interpretation of “long shot”) and say the prior is 1/100 – a 1% chance that fluvoxamine truly is effective in the treatment of COVID-19 patients with mild symptoms. Remember, going into this trial, all that exists is some scant laboratory data in small numbers of rats and mice in an inflammatory disease that is not COVID-19.
So, now we apply my favorite formula for combining prior knowledge/belief with current data to produce a posterior probability of the alternative hypothesis being true – i.e., fluvoxamine works. This has been widely discussed in other Blogs on this site (see for example Blog #7). Stealing from Blog #7 is the following:
Using a point prior, there is a simple approximation for computing the probability of H0 being false using the Bayes Factor Bound (BFB), which is based on reasonable, practical assumptions. Let p0 be the prior probability that H0 is false and let p=p-value from the test of H0 from the current experiment. Then the Bayes Factor Bound is
BFB=1/[-e*p*ln(p)],
and the upper bound on the posterior probability that H0 is false (p1) given the observed data is
p1 ≤ p0*BFB/(1-p0+p0*BFB) (Equation 1).
In our case, p0 = 0.01 (my prior) and p=0.009 (the current evidence). For the upper bound on the posterior probability that fluvoxamine works, given the results of the 152 patient study conducted by Dr. Lenze and colleagues, is p1≤0.081. That is, there is at most an 8% chance that fluvoxamine really works. That’s really small in my mind, and not nearly as convincing as merely looking at the p-value of 0.009 or the statement by Dr. Lenze, “So the results were really pretty incredible.” To be sure, moving the evidentiary needle from 0.01 (prior) to 0.08 (posterior) is a substantial jump based on one small trial that produced some good evidence of effectiveness. It is just that the starting point is quite low, and we should admit to the possibility that the initial study by Dr. Lenze et al. is a false positive finding – just like false positive findings with diagnostics tests (see Blog #6).
If the prior was really a “long shot,” say 1/1000, then with a p-value of 0.009, the posterior probability would have an upper bound of 0.1%, an exceedingly small number. In order for the posterior probability to be a 50/50 proposition, that is a 50% probability that fluvoxamine truly works, the prior would have to be 1/10. Some might argue for such a prior, but that is too high in my mind given the rationale presented in this section.
So, my answer to the second question, “What do you believe?” is that there is a pretty low probability that fluvoxamine will work in the ongoing, larger trial. That study, entitled “Fluvoxamine for Early Treatment of Covid-19 (Stop Covid 2)” plans to randomize 880 patients and is scheduled to complete by late summer of 2021. See ClinicalTrials.gov (https://clinicaltrials.gov/ct2/show/NCT04668950?term=fluvoxamine&cond=Covid19&draw=2).
What do I decide?
For Dr. Lenze and team the answer comes from his 60 Minutes interview. Dr. Eric Lenze stated, “I have to be a scientist about this. We’ve tested it in one study. But– in my view, it needs to be confirmed in a larger study.”
With the pandemic still being a global problem, and vaccinations globally being a long time off despite progress in the most developed countries, and the need for a treatment for patients with mild symptoms (antibody treatments are geared for those more seriously ill in order to prevent death, ventilation or shorten the length of hospital stay), it’s probably reasonable to pursue fluvoxamine in the larger trial. However, investors, scientists and others should proceed with eyes wide open as to the likelihood of success.
For any decisions, one must assess the probability that the first trial by Dr. Lenze is a false positive finding and the ensuing cost of subsequent trials that have little chance of succeeding (e.g., like the 8% I am using as my estimate). One has to balance that cost versus the cost of NOT going forward with the next study and leaving a potentially effective treatment for mild COVID-19 on the shelf to the detriment of thousands and perhaps millions of patients. In the middle of this crisis, it seems reasonable to leave no stone unturned and therefore to proceed with the trial. Are you sure? Such a decision does come with more than just the cost of doing the study. The level of confusion, distraction, splintered resources, studies competing for patients, etc. are all an enormous drain on the medical/scientific community. In some cases, ineffective but hyped treatments are used by physicians or taken by unsuspecting patients with fatal consequences. Ivermectin is a very recent example. [7]
So, what would I decide? I do feel better about the subsequent trial of fluvoxamine given the “Phase 2” trial was a randomized, double-blind, placebo-controlled trial, albeit with the caveats mentioned already. Given the unmet medical need and the size of that medical need, I concur that the “Phase 3” trial should proceed. I am a bit disappointed that the study is still a completely remote study with no patient-physician interaction and all evaluations being self-administered by the patient in their home.
As a betting man, I would place my bet on the study not being successful just as many other once-promising interventions for COVID-19 (e.g. convalescent plasma) have proven ineffective when subjected to the rigors of a large scale clinical trial of meaningful clinical outcomes. This might seem contradictory to the casual reader: he wants the next trial to proceed, but he doesn’t think it will succeed. I think this is a rational course of action: the cost of failing in the next trial is smaller than the societal cost of having an effective treatment that never gets its chance in a big “Phase 3” trial. It’s like sending in a pinch-hitter to the plate in the bottom of the 9th and needing a home run to win the game. It is doubtful that the pinch hitter can pull off a home run, but I would rather “go down swinging” in this case.
Finally, as with all probabilities, we will never know if I am right or wrong with my probability calculation. The drug will either succeed or fail in the one Phase 3 trial that is done. If it fails, then I am right; after all I did say there was a 92% chance of failure. If the trial succeeds, then I am also right; I did say there was an outside chance of success! Just like the pinch hitter who has an 8% chance of hitting a home run in the bottom of the 9th inning to win the game. He may hit the home run, or he may not. It’s one observation form which we cannot validate a probability calculation. That only can happen over repeated trials under the same or similar circumstances. That’s probability. It’s not black and white.
I will anxiously await the Phase 3 results of fluvoxamine. I truly hope my prediction is wrong and we have a new treatment for a population of mild patients that is in great need. Lots of breakthroughs in science occurred by betting against the odds.
References
[1] Lenze, E. J., et al., Fluvoxamine vs Placebo and Clinical Deterioration in Outpatients With Symptomatic COVID-19: A Randomized Clinical Trial. JAMA. 2020;324(22):2292-2300.
[2] Seymour, C. W. et al., COVID-19 Infection—Preventing Clinical Deterioration. JAMA. 2020;324(22):2300.
60 Minutes Transcript – https://www.cbsnews.com/news/fluvoxamine-antidepressant-drug-covid-treatment-60-minutes-2021-03-07/
[4] Ishima T, Fujita Y, Hashimoto K. Interaction of new antidepressants with sigma-1 receptor chaperones and their potentiation of neurite outgrowth in PC12 cells. Eur J Pharmacol. 2014;727: 167-173.
[4] Rosen DA, Seki SM, Fernández-Castañeda A, et al. Modulation of the sigma-1 receptor-IRE1 pathway is beneficial in preclinical models of inflammation and sepsis. Sci Transl Med. 2019;11 (478):eaau5266.
[5] Ishima T, Fujita Y, Hashimoto K. Interaction of new antidepressants with sigma-1 receptor chaperones and their potentiation of neurite outgrowth in PC12 cells. Eur J Pharmacol. 2014;727: 167-173.
[6] Califf, R. M. et al. Weighing the Benefits and Risks of Proliferating Observational Treatment Assessments: Observational Cacophony, Randomized Harmony. JAMA August 18, 2020 Volume 324, Number 7, 625-626.
[7] Lopex-Madira et al. Effect of Ivermectin on Time to Resolution of Symptoms Among Adults With Mild COVID-19: A Randomized Clinical Trial. JAMA. doi:10.1001/jama.2021.3071 (Published online March 4, 2021).
analytixthinking.blog 