Blog 26 – Estimands Part 3A – What Is the Question?

Acknowledgement: I have used the pronoun “I” throughout this Blog since it is my Blog and I take responsibility for its content. However, what follows is the result of extensive discussions with Dr. Carl Peck, MD, Adjunct Professor in the School of Pharmacy, University of California San Francisco, Founder and Chairman, NDA Partners, and former Director of the Center for Drug Evaluation and Research (CDER) at FDA. As noted by his credentials, he has vast experience in clinical trials from regulatory, academic and industry perspectives. He has contributed to reviewing and editing this Blog as well as writing some elements of it.

Introduction

Now that you have read Blog 24 (some background in ITT) and Blog 25 (some background on what is an estimand), we are ready to tackle a review of the paper that instigated this set of Blogs, “A Perspective on the Appropriate Implementation of ICH E9(R1) Addendum Strategies for Handling Intercurrent Events.” [Fleming et al, 2025], henceforth referenced merely as Fleming. The claims put forth in that paper can be summarized using the language of their Abstract.

• Fleming argues that some strategies described in the ICH E9(R1) Guideline “don’t preserve integrity of randomization” and “don’t reliably capture the intervention’s meaningful net effects. These approaches have inherent limitations in ability to draw scientifically rigorous inference on clinically relevant causal effects important for decisions about adopting interventions.”
• They also contend that ITT-based estimands are “relevant to real-world settings because they are unconditional in nature i.e., they don’t condition on post-treatment circumstances that might not be many patients’ experiences. … and [don’t] properly evaluate the experimental intervention within a regimen that includes possible ancillary care.”
• Furthermore, they advocate defining estimands with standard of care (SoC) control arms using treatment policy (with composite strategy) to “achieve scientifically rigorous causal inference.”

There are good and useful perspectives in the Fleming paper, but the key to this blog, and indeed ICH E9(R1), henceforth the Addendum [ICH, 2019], is that these are not the only scientifically valid and clinically meaningful perspectives that may be relevant to designing, analyzing and interpreting results from a clinical trial. The purpose here is to expand on those perspectives and provide some additional, clinically meaningful points to consider. The intent herein is thus not to comment on all aspects of the Fleming paper. The focus will be on two prominent recommendations from their paper: (1) the use of intention-to-treat (ITT) as an analysis approach, which will be covered in this Blogs 26 and Blog 27 (forthcoming), and (2) the employment of a standard of care (SoC) for the control arm, which will be covered in Blog 28 (forthcoming), bearing in mind that we are seeking to answer the fundamental drug development question, “Does this experimental treatment cause that outcome?” I have divided the review into multiple Blogs so that they can be read in a modest timeframe and are more easily digestible chunks of information. With that said, this Blog is a little longer since it is at the heart of estimand thinking.

Strap on your seatbelts. Be ready and open your mind to a newer perspective – one that expands on and clarifies ICH E9(R1). Then ask yourself, “What is the question?”

Understanding and Defining the Treatment Effect

A primary motivator for creating the Addendum is revealed in the concept paper that was used to gain the approval of the International Council for Harmonization (ICH) for further study and development of a formal guideline [ICH, 2014]. Essentially, the estimand is the “golden thread” that ties the clinical question of interest with the clinical trial objective regarding a treatment effect and associated clinical trial design, statistical analysis and interpretation of trial results. The authors of the Addendum recognized that too often debates occurred regarding the appropriate analysis of a clinical trial, with less focus on the clinical question regarding the most important treatment effect. In short, there had been too much focus on the “how” and not enough on the “what.” If researchers cannot agree on the what, they will never agree on the how. So, the Addendum, in large part, was meant to “flip the script” and first focus on what clinical question(s) is of interest – and thereby treatment effect – followed by considerations for how best to answer that question through trial design, data collection and statistical analysis. First WHAT; then HOW.

Intention-to-Treat

In Fleming, Section 2 proposes “specific elements of design, conduct, and analysis which enable a trial to provide scientifically rigorous and reliable causal inference on clinically meaningful estimands.” Five elements are presented with the fifth element being a “Primary analysis of trial data based on the ITT principle.” There is additional elaboration that describes the usual notions of ITT – analysis of all randomized patients and inclusion of outcomes on all such trial subjects at the endpoint of the trial. The use of a treatment policy strategy (as described in the Addendum) ignores intercurrent events (as described in the Addendum). Fleming notes, “This is commonly referred to as an ‘intention-to-treat (ITT)’ analysis.” We will investigate this thinking more deeply through the lens of the Addendum.

The opening sentences of the Addendum states, “To properly inform decision making by pharmaceutical companies, regulators, patients, physicians and other stakeholders, clear descriptions of the benefits and risks of a treatment (medicine) for a given medical condition should be made available. Without such clarity, there is a concern that the reported ‘treatment effect’ will be misunderstood.” Defining a treatment effect is arguably the predominant theme of the Addendum.

If we want to be clear and precise with what we mean by a treatment effect, then we must be clear and precise about what we mean by “treatment” and “effect.”

First WHAT, then HOW.

Precisely Defining the Treatment Condition

The Addendum addresses the notion of “treatment” as an essential attribute of an estimand. Initially, the Addendum describes this attribute as the “treatment condition” and elaborates on the matter by describing that the treatment condition could involve a variety of dosing schemes or concomitant therapies in conjunction with the experimental medication. Unfortunately, the phrase “treatment condition” is abbreviated to “treatment” throughout the Addendum, and in my experience, many applications of the estimand framework simply refer to treatment as the experimental medication rather than a more comprehensive notion of treatment condition. Thus, when other medical interventions occur during the course of the trial – e.g., changes in dose of the experimental medication, use of rescue medications, changes in concurrent medications, hospitalization, ventilation, surgery, or any other medical care that is needed for the patient  – they are viewed as intercurrent events. In the perspective of Fleming, these should not be considered as intercurrent events in that a treatment policy strategy should be applied with all patients followed and outcomes measured regardless of the standard medical care of the patient, which is consistent with ICH E9 and its definition and application of ITT. [ICH, 1998] In that sense, the various interventions that represent the standard of care (SoC) for the patients are not really intercurrent events at all but rather part of the treatment condition of interest as contemplated by the Addendum.

Thus, in Fleming, the treatment condition of interest, in the parlance of the Addendum, is the initiation of the randomized study medication in the context of the current standard-of-care. Once a patient is initiated on the randomized study medication, they can never be uninitiated, and therefore, all events and interventions that occur after initiation are included in the assessment of the initiation effect. The logical conclusion is to use an ITT analysis approach. However, many stakeholders, including regulators, are very interested, perhaps primarily interested, in the direct effects (beneficial and adverse) of the new experimental treatment, independent of substantial confounding by other medical interventions. Thus, in the Addendum, such medical interventions represent potential intercurrent events.

Little and Kang [2015] have discussed ITT analysis and provide critical insights into the advantages and disadvantages of its use. When employing an ITT analysis, they note, “The disadvantage is that the treatment effect may include the effects of treatments other than the treatment under study, so the study is in effect assessing a ‘treatment regimen’ that involves these other treatments. In many studies, the main interest is not a treatment regimen but rather in the particular effects of the new treatment.” I agree with Little and Kang. The Addendum notes this perspective in its Introduction: “It remains undisputed that randomisation is a cornerstone of controlled clinical trials and that analysis should aim at exploiting the advantages of randomisation to the greatest extent possible. However, the question remains whether estimating an effect in accordance with the ITT principle always represents the treatment effect of greatest relevance to regulatory and clinical decision making.”

Furthermore, I also agree with the Addendum on fully defining the treatment condition of interest, which in practice, requires considerable attention.

Thus, a clear definition of what is meant by “treatment” is required. Is it

• the experimental medication alone, or
• the experimental medication as a combination therapy, or
• a well-defined treatment policy (e.g., use second-line treatment Z following disease progression or failure on the randomized study medication),
• the experimental medication in the context of any and all other care provided to the patient,

or many other descriptions of possible treatment conditions?

Clearly Defining the Treatment Effect – An Example

Along with clearly defining what is meant by the treatment condition, there must be absolute clarity as to the treatment effect question of interest when precisely defining an estimand. While the Addendum may contain some tacit notions about the treatment effect, it is not explicitly stated as a major consideration of the estimand framework. For clarity, we propose that there are two distinct treatment effect questions that are clinically meaningful [see also Hernan and Hernandez-Diaz. 2012]:

• Does the initiation of this treatment (in the context of all other care for the patient) cause the outcome? and
• Does the taking of this treatment as intended or defined by the protocol cause the outcome?

I believe that these are the only two treatment effect questions; there is nothing in between. The perspective in Fleming focuses on the first treatment effect question and the use of an ITT analysis, and furthermore, states or implies that this is the only valid scientific and clinically meaningful question. I disagree.

An excellent example to illustrate the distinction between these two treatment effect questions is the study of colonoscopy screening for the prevention of colon cancer and death [Bretthauer et al, 2022]. In that study, 84,585 participants from Poland, Norway and Sweden were randomized 1:2 to receive an invitation to obtain a screening colonoscopy versus no invitation for a screening colonoscopy. The primary outcome measures were the risks of colorectal cancer and related deaths and all-cause mortality after a 10-year follow-up period with nearly complete outcome data on all randomized patients. The primary analysis used the ITT approach and reported a relative risk (RR) for colorectal cancer of 0.82 [95% CI: (0.70, 0.93)], and a RR for all-cause mortality of 0.99 [95% CI: (0.96, 1.04)]. Thus, the interpretation of the study was that there was a moderate effect of colonoscopy screening on reducing the risk of colorectal cancer but no effect on all-cause mortality. Bloomberg News reported on the study using the headline “Screening Procedure Fails to Prevent Colon Cancer Deaths in Large Study.” Other broadcast and print news agencies ran stories or articles providing a skeptical view of the value of colonoscopy screening based on the study results.

A closer inspection of the study reveals that 28,220 patients received an invitation for colonoscopy screening and 56,365 received no invitation. Of those in the invited group, 11,843 (42%) actually received a screening colonoscopy while in the no-invitation group, the authors reported “no opportunistic screening of any meaningful extent.” When a protocol-defined supplemental analysis was done considering the direct effect of undergoing the colonoscopy screening procedure, the results were markedly different. The relative risk of colorectal cancer was 0.69 [95% CI: (0.55, 0.83)], and the relative risk for all-cause mortality was 0.50 [95% CI: (0.27, 0.77)], which are markedly different than the ITT analysis results. One can debate the credibility of an analysis that adjusts for the potential selection bias related to those who accepted the invitation for a screening colonoscopy; however, the authors provide good supporting evidence to justify these effect estimates of actually receiving the intervention of interest – a screening colonoscopy. I believe this estimand (the direct effect of receiving a screening colonoscopy) is clinically meaningful and the estimate of that effect is scientifically valid., i.e., a credible answer to the clinical question, “Does screening colonoscopy cause a reduction in all-cause mortality?” In fact, the supplemental analysis of the effect of receiving a screening colonoscopy is the scientific question, while the ITT effect related to the invitation for a screening colonoscopy is a behavioral/societal question.

This study and its reporting highlight two important considerations relevant to this commentary. First, what is the question? Are we interested only in the effect of the invitation to receive a screening colonoscopy or the clinically meaningful effect of receiving a screening colonoscopy? These are very different questions with very different answers. The first question and its ITT estimate answers are of significant interest to public health policy. When offered an invitation to a screening colonoscopy, what is the overall outcome in the patient population being studied? This is important to know. The study results were moderately positive but not compelling, especially when considering the more important outcome – mortality. The answer to the second question is of paramount medical and scientific interest. When a screening colonoscopy is performed, what is the clinical outcome in those patients? The results were dramatically positive, especially regarding mortality. So, one could interpret the study overall to say that screening colonoscopy is extremely effective at preventing colorectal cancer and all-cause mortality, but we need much better methods for moving the patient from invitation to actuation of colonoscopy screening.

The second consideration concerns the use of precise, scientific language to describe the effect being estimated. The colonoscopy screening study was designed to test the effect of the invitation to colonoscopy screening. That’s how patients were randomized and observed. That’s how the ITT analysis was applied. Yet the title of the article, and in many statements throughout the article, the authors use the phrase “the effect of colonoscopy screening.” This is inaccurate and misleading. However, it is done quite often; an ITT analysis is performed to estimate the effect of a treatment strategy involving the initiation of an experimental treatment, but the results are communicated as the direct effect of the experimental treatment. This is prevalent in medical publication and can also be seen in regulatory labels of approved medications. This is scientifically inaccurate and in the extreme cases, disingenuous and potentially harmful, as with the colonoscopy screening study. Both clinical questions  – the public health effect and medical/scientific effect – from the colonoscopy screening study are important. Absolute clarity for describing the treatment effect question and its answer is critical, lest there be confusion about the study findings.

Lastly, the use of the ITT approach can dilute the direct treatment effect or create a so-called “bias towards the null.” Some see this as a value of ITT in that it is conservative, though that is not always the case [Hernan and Hernandez-Diaz. 2012]. If an experimental treatment can produce a statistically significant result when using the ITT approach, then surely the experimental treatment must work on its own. However, this logic comes with costs. First, sample sizes are generally increased to accommodate bias toward the null, resulting in increased time and cost of studies, as well as exposure of more patients to potentially ineffective treatments. Second, sponsors tend to use larger doses of the experimental treatment in the hopes of producing a larger treatment effect on efficacy to offset the effect dilution using the ITT approach. Regularity approval of a new drug based on doses employed in trials that are higher than necessary can lead to exaggerated efficacy or an increased risk of adverse reactions for a fully adherent patient. These unintended consequences of ITT related doses may account in part for the increasing frequency of post-approval, safety-related dose reductions: 20% during 1980-1999 [Cross et al, 2002], 40 % during 2000-2017 [Ashida et al, 2021], and the relatively constant frequency of safety-related market withdrawals in recent decades [Lasser et al, 2002; Philipson et al, 2008]. 

The Effect of Taking a Treatment

I believe there are credible ways to answer the clinically meaningful question, “What is the effect of taking this treatment as intended?” Long before the comments of Little and Kang [2015] regarding the importance of assessing the direct effect of a new medication, many have presented arguments in favor of understanding the effect of a treatment on patients who can adhere (comply) to the treatment condition (per the Addendum) as well as making regulatory decisions about its approval and ultimate use by prescribers and patients [Hernan and Hernandez-Diaz. 2012; Sheiner and Rubin, 1995; Sheiner, 2002; Hernan and Robins, 2017].

Evaluating the effect of taking a treatment requires further elaboration depending on the clinical context. There are three possible refinements relating to three clinically meaningful questions. What is the treatment effect …

1. If all patients could take the experimental treatment as prescribed?
2. While patients take the experimental treatment as prescribed?
3. In those who can adhere to the experimental treatment as prescribed?

Though not explicitly stated in the Addendum, the first relates to the hypothetical strategy, the second relates to the while on treatment strategy, and the third is a principal stratum of adherers related to the principal stratum strategy..

The hypothetical approach as described in the Addendum is discouraged, even dismissed, by Fleming. I agree that special care is needed when considering such an approach, especially when, as often happens, researchers actually make counterfactual assumptions rather than hypothetical assumptions. I won’t delve into the philosophy of hypothetical versus counterfactual, but it is covered in my book, Does This Treatment Cause That Outcome? [Ruberg, 2026] In short, for example, assessing a treatment effect as if all patient could take the medication as prescribed for the duration of the trial, when in fact some patients have adverse event that prevent them from taking the medication, is a hypothetical that has little, or perhaps no, clinical relevance. It is actually a counterfactual question (i.e., counter to the observed facts). However, as in the screening colonoscopy study, the authors note, “We estimated … the effect of screening if all the participants who were randomly assigned to the invited group had undergone screening.” I believe this is a very scientific and clinically meaningful hypothetical question worthy of answering with the best statistical methods available, as was done in the paper.

The “while on treatment” strategy is also dismissed by Fleming but -much like the ITT approach – it uses all randomized patients and has a measure of the treatment effect at the end of their treatment period. Of course, in this scenario, the end of the treatment period is different for each patient and may be shortened due to adverse events or lack of efficacy, among other reasons. Nonetheless, it is closely aligned with the ITT approach and should not be so easily dismissed. In chronic progressive diseases (e.g., Alzheimer’s, COPD, some cancers), patients may proceed through a sequence of treatments – including some trial and error – and it is a clinically meaningful question to ask, “What is the effect of the treatment while I am taking it?” which may last for 6 months or for 3 years. Will it cure the patient? Halt the progression of the disease? ? Slow the progression of the disease? Of course, such information should be accompanied by the expected duration of the effect. Then patients and physicians can weigh the benefits and duration and risks of the treatment for their particular circumstances.

As Akacha et al [2017] describe, many patients want an answer to the question, “If I can take this treatment as prescribed, what can I expect to happen?” with an implicit considerations for both efficacy and safety. This principal stratum approach is also dismissed by Fleming. Patients recognize that they may have an adverse event when taking a new treatment, and they recognize that not all drugs work for all patients. But “if I can take this treatment,” what are the upsides and downsides? Now, a common criticism of providing an estimate of the treatment effect in the principal stratum of adherent patients is that, at the point of prescribing, we never know who will adhere; that is, who is in that principal stratum? Thus, such estimate should not be used for decision-making … whether it be for regulatory approval of medical practice. I strongly disagree.

Patients are often told about possible side-effects of a treatment, and at the point of prescribing, we do not know whether they are in the principal stratum of patients who will have such an adverse event. It is still very important information to convey, and information that can factor in decision-making. Furthermore, there may be long-term side effects of a treatment, and it is equally important to convey that, in the principal stratum of patients who adhere to the treatment for whatever duration is relevant for the disease and treatment, there are such long-term side effects. Again, this is important information for both regulatory and clinical decision-making. Such conditional statements are criticized in Fleming but are very real considerations in decision-making at all levels – at the population level for regulatory decision-making and at the individual level for prescribing decisions.

I am a strong supporter of randomization for rigorous clinical trials. Randomization is an essential  tool for designing and executing clinical trials. Maintaining the integrity of randomization in the analysis is also very important, but not more important than answering the clinical question of primary clinical interest whether it be for regulatory decision-making or clinical practice. The statistical “tail” should not wag the clinical “dog.” Even in the absence of an ITT approach, randomization is useful for performing analyses that answer other clinical questions besides the ITT effect (e.g., while-on-treatment, as well as building models for the adherers average causal effect, AdACE). [Qu et al, 2021] First WHAT, then HOW.

Finally, if you follow Akacha et al [2017] and believe that principal stratification can be used to answer both regulatory approval and clinical practice questions, then the Tripartite Estimand Approach (TEA) is a very clinically meaningful approach for describing a treatment’s effects (note the plural). It’s what (first WHAT) patients and physicians want to know. The TEA  proposes treatment effect estimands for:

• The probability of discontinuing the treatment due to an adverse event
• The probability of discontinuing the treatment due to lack of efficacy
• The treatment effect estimate for the principal stratum of patients who would adhere to the randomized study medications.

To borrow a phrase from Fleming, one could argue that these “capture the intervention’s meaningful net effects” (note the plural).

The TEA provides estimates at the population level for the effects (plural) of the treatment and conveys information for balancing benefit-risk decision-making by the prescriber at the individual patient level. For example, a physician speaking to a patient might say, “If you can adhere to this medication as prescribed, then you can expect a decrease in HbA1c of 1.5%. About 80% of patients are able to adhere based on clinical trial results. If you are one of the 20% who cannot adhere for whatever reason, we can consider other approaches and medications to lower your HbA1c.” Additional clarity can be provided by describing the proportion of patients who discontinue the medication due to adverse events and the nature of such adverse events. Similar statement can be imagined for a wide variety of disease states and outcomes. Such conditional statements are clearly of interest to patients [Akacha et al, 2017; Qu et al, 2021]. The TEA is covered thoroughly in Ruberg [2026].

Summary

I find it both interesting and telling that the Fleming et al paper, as indicated in the title, focuses on strategies for handling intercurrent events. This is consistent with my observation that statisticians focus too much, and sometimes primarily, on the HOW and not the WHAT – the clinical question of primary interest.

The Addendum is written in the context of drug development as a guidance to sponsors seeking regulatory approval of a new medication by seeking “to establish the existence, and to estimate the magnitude, of treatment effects.” I have emphasized the need to define precisely what is meant by “treatment,” and what is meant by “effect.” I have presented the notion that there are only two clinical/treatment effect questions of interest:

• Does the initiation of this treatment (in the context of other care for the patient) cause that outcome, and
• Does the taking of this treatment as intended or defined by the protocol cause that outcome?

For the effect of initiating a treatment, there are no difficulties in defining the treatment attribute of the estimand. The treatment condition, as described in the Addendum, is simply the initiation of the experimental treatment (taking at least one dose). Post-initiation interventions do not constitute intercurrent events. When “terminal” events are expected (e.g., death, transplant, amputation), the estimand can be prospectively defined to accommodate such events, as suggested in Fleming, once again obviating the purported intercurrent event. Taking an ITT analysis approach (the HOW) is completely appropriate for estimating this effect (the WHAT). This is the approach espoused by Fleming.

As noted by Hernan and Hernandez-Diaz, “An ITT analysis of RCTs is appealing for the same reason it may be appalling: simplicity.” [2012] But rather than debate its appeal or scientific validity, we should be debating what is the clinical question of interest. If that question is the effect of initiation of a treatment, then the ITT approach is wholly warranted. If we want to estimate the direct effect of the experimental treatment (recall the screening colonoscopy study), then an ITT analysis is biased, and we do not know the direction of the bias. [Hernan et al, 2013] See also Blog 27 upcoming.

For estimating the direct effect of taking the treatment as intended, greater consideration is required for defining the treatment condition. Besides the experimental medication, one must consider background therapies and possible rescue medications as well as the timing and duration of such medicinal interventions. Does the treatment condition include the experimental medication with stable background therapy, and if so, what constitutes “stable”? There are many other such considerations, and the Addendum provides some additional examples. Again, Ruberg [2026] provides a thorough review and discussion with many examples.

In any case, once the complete treatment condition for the estimand is precisely defined, which I describe as the estimand-defined study treatment (EDST) in my aforementioned book, one can define clearly whether a patient is adherent to the EDST or not. There is nothing in between. Thus, the only intercurrent event of interest is the discontinuation of the EDST (though there may be many reasons for such discontinuation), and observations after the cessation of the EDST (i.e., the complete treatment condition description) are not relevant for estimating the effect of taking (or adhering to) the EDST. (Note: some observations that are made within a short timeframe after cessation of the estimand-defined study treatment may be included in the analysis since there is likely some waning of the treatment effect that is specific to the experimental medication and disease state).

With this clarity, one can choose a strategy per the Addendum (hypothetical, while on treatment, principal stratum) for how to handle patients who have taken the EDST to varying degrees, including analysis of the principal stratum of patients who would adhere to both the experimental medication and the control. These strategies require assumptions and statistical modeling, but that does not make them scientifically invalid or clinically meaningless. There are statistical approaches that provide an answer, perhaps approximate, to the second clinically meaningful question noted above.

To paraphrase John Tukey [1962], an approximate answer to the right question is better that a precise answer to the wrong question. We must start with the right clinical question for the clinical objective, the disease condition and patient population of interest. We have proposed two fundamental causal questions about a treatment effect. We should align our statistical approach with providing the best, even if approximate, answer to the most relevant question. First WHAT, then HOW.

References

Fleming TR, Carroll KJ, Wittes JT, Emerson SS, Rothmann MD, Collins S, Levin G. A Perspective on the Appropriate Implementation of ICH E9(R1) Addendum Strategies for Handling Intercurrent Events. Stat Med. 2025 May;44(10-12):e70104. doi: 10.1002/sim.70104.

International Council for Harmonization (2019), “E9(R1): Addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials R9(R1),” available at https://database.ich.org/sites/default/files/E9-R1_Step4_Guideline_2019_1203.pdf.

International Council for Harmonization (2014), “Final Concept Paper E9(R1): Addendum to Statistical Principles for Clinical Trials on Choosing Appropriate Estimands and Defining Sensitivity Analyses in Clinical Trials,” available at https://database.ich.org/sites/default/files/E9-R1_EWG_Concept_Paper.pdf.

International Council for Harmonization (1998), “Statistical Principles for Clinical Trials – E9.” Available at https://database.ich.org/sites/default/files/E9_Guideline.pdf.

Little R, Kang S. Intention-to-treat analysis with treatment discontinuation and missing data in clinical trials. Stat Med. 2015;34(16):2381-90.

Hernan MA, and Hernandez-Diaz S. Beyond the intention to treat in comparative effectiveness. Research. Clin Trials. 2012; 9(1): 48–55.

Bretthauer M, Løberg M, Weiszczy P, et al. Effect of Colonoscopy Screening on Risks of Colorectal Cancer and Related Death, New Eng J Med. 2022;387(17): 1547-56.

Cross J, Lee H, Westelinck A, Nelson J, Grudzinskas C, Peck C.  Postmarketing drug dosage changes of 499 FDA-approved new molecular entities, 1980-1999. Pharmacoepi Drug Safety. 2002;11(6):439-446.

Ashida M, Narukawa M.  Post-Marketing Change in Dosage and Administrations of FDA-Approved Drugs Between 2000 and 2017. Adv Pharmacoepi and Drug Safety.2021;10:1-5.

Lasser KE, Allen PD, Woolhandler SJ, Himmelstein DU, Wolfe SM, Bor DH. Timing of New Black Box Warnings and Withdrawals for Prescription Medications. JAMA. 2002;287(17):2215–2220.

Philipson T, Berndt ER, Gottschalk AHB, Sun E. Cost-benefit analysis of the FDA: The case of the prescription drug user fee acts. 2008;92(5-6):1306-1325.

Sheiner LB, Rubin DB. Intention-to-treat analysis and the goals of clinical trials. Clin Pharmacol Ther. 1995;57(1):6-15.

Sheiner LB. Is intent-to-treat analysis always (ever) enough? Br J Clin Pharmacol. 2002;54(2):203-11.

Hernan MA,  and Robins JM. Per-Protocol Analyses of Pragmatic Trials. N Engl J Med 2017; 377(14):1391-1398.

Ruberg, SJ. Does This Treatment Cause That Outcome? The Science of Estimating a Treatment Effect and Why It Matters. CRC Press, Boca Raton, 2026.

Akacha M, Bretz F, Ruberg S. Estimands in clinical trials – broadening the perspective. Stat Med. 2017;36(1):5-19.

Qu Y, Luo J, Ruberg SJ. Implementation of tripartite estimands using adherence causal estimators under the causal inference framework. Pharm Stat. 2021;20(1):55-67.

Hernan MA, Hernandez-Diaz S, Robins JM. Randomized Trials Analyzed as Observational Studies. Ann Intern Med. 2013;159:560-562. Tukey J. The Future of Data