Missing data is a common challenge in clinical trials. Participants may discontinue treatment, miss assessments, withdraw consent, or become lost to follow-up. When outcomes are unavailable, their effect on the estimated treatment benefit cannot be assessed directly.
Statistical methods can account for incomplete data, but their validity depends on assumptions about the missing outcomes. Since these assumptions usually cannot be verified using the observed data alone, sensitivity analyses are needed to examine whether the study conclusions remain robust under alternative, clinically plausible scenarios.
Tipping point analysis tests how far assumptions about missing outcomes must change before a predefined study conclusion changes.
Tipping point analysis is a missing data sensitivity analysis that systematically varies assumptions about unobserved outcomes until a predefined study conclusion changes.
Depending on the study objective, that change might occur when:
The scenario under which the conclusion changes is known as the ‘tipping point’. The analysis assesses how dependent the study conclusion is on assumptions that cannot be confirmed from the observed data.
Missing data can introduce bias, reduce precision, and make treatment effects more difficult to interpret. The risk is particularly important when participants with incomplete data differ systematically from those who complete the study.
For example, participants may discontinue because their symptoms worsen, treatment is ineffective, or adverse events occur. If the reasons for missingness are associated with the unobserved outcome, an analysis based only on the observed data may not provide a complete picture.
Many primary analyses assume that data are Missing At Random (MAR). This can be reasonable when the probability of missingness can be explained by observed information included in the model. However, uncertainty remains when missing outcomes may be systematically better or worse than predicted under MAR.
ICH E9(R1) places sensitivity analysis within a framework connecting the trial objective, estimand, main estimator, and assumptions underlying estimation. A sensitivity analysis should therefore target the same estimand as the main analysis while assessing the strength of its assumptions.
Missing data mechanisms are commonly described as Missing Completely At Random (MCAR), Missing At Random (MAR), or Missing Not At Random (MNAR). These classifications describe the assumptions made about the relationship between missingness and the study data.
MCAR assumes that missingness is unrelated to observed or unobserved data. For example, an assessment might be unavailable because a laboratory sample was damaged during transport. The reason for its absence is unrelated to the participant’s characteristics or underlying outcome.
MCAR is useful conceptually, but the assumption may be difficult to justify for many forms of missing clinical trial data.
MAR assumes that, after accounting for observed information, missingness does not depend on the unobserved outcome. For example, participants with greater baseline disease severity may be more likely to miss later visits. If baseline severity and other relevant predictors are included in the model, the missing outcomes may be estimated using the observed information.
Multiple imputation and mixed models commonly rely on an MAR assumption, although their precise assumptions depend on the model and implementation.
MNAR means that the probability of missingness may still depend on the unobserved outcome after accounting for available data. For example, participants whose symptoms worsen may be more likely to discontinue before the final assessment. Their missing outcomes may therefore be worse than the values predicted under MAR.
The observed data alone generally cannot distinguish definitively between MAR and MNAR. Tipping point analysis explores departures from the primary missing-data assumption without claiming to establish the true missing-data mechanism.
Tipping point analysis is most useful when uncertainty about missing data could materially affect an important clinical trial conclusion.
Relevant situations can include:
The EMA guideline on missing data in confirmatory clinical trials addresses the effect of missing values on bias, variability, interpretation, and regulatory review. It also includes recommendations concerning sensitivity analyses and the prespecification of missing data methods.
Tipping point analysis is not automatically required for every endpoint. Its use should be driven by the clinical question, estimand, endpoint, missing-data pattern, reasons for missingness, and primary analysis strategy.
For a continuous endpoint, tipping point analysis often begins with a multiple-imputation analysis conducted under an MAR assumption. A delta adjustment is then applied to imputed, but not observed, values to represent departures from MAR.
The primary analysis is performed using the method predefined in the protocol or Statistical Analysis Plan (SAP).
Under a multiple-imputation approach, several complete datasets are generated by replacing each missing value with a draw from an appropriate predictive distribution. Each dataset is analysed using the predefined analysis model. The resulting estimates are commonly combined using Rubin’s Rules to account for both within-imputation and between-imputation variability.
A user-defined shift, usually called a delta adjustment, is specified to represent a departure from the MAR-based predictions.
For example, if a higher endpoint value indicates improvement, negative delta values could be applied to missing outcomes in the active-treatment group:
|
Scenario |
Delta applied to imputed values |
|
Primary MAR analysis |
0 |
|
Scenario 1 |
−1 |
|
Scenario 2 |
−2 |
|
Scenario 3 |
−3 |
|
Scenario 4 |
−4 |
The direction of the shift must reflect the endpoint definition. For an endpoint where lower values represent improvement, the direction needed to create a less favourable outcome would be reversed.
The selected delta is added to the imputed values while observed outcomes remain unchanged. The analysis is repeated over a range of delta values. The analysis model should generally remain consistent with the primary analysis so that changes in the result can be attributed to the alternative missing-data assumptions rather than a different statistical model.
Each adjusted imputed dataset is analysed separately. Treatment-effect estimates, standard errors, confidence intervals, and p-values are then combined using the prespecified pooling method. This process is repeated for each delta scenario.
The tipping point is the first evaluated scenario under which the predefined study conclusion is no longer supported. Consider the following illustrative results:
|
Delta applied |
Treatment difference |
P-value |
Statistically significant? |
|
0 |
−1.8 |
0.001 |
Yes |
|
−1 |
−1.5 |
0.008 |
Yes |
|
−2 |
−1.2 |
0.030 |
Yes |
|
−3 |
−0.9 |
0.070 |
No |
In this example, the result changes between delta values of −2 and −3. The next step is to determine whether a deterioration of this magnitude among participants with missing outcomes is clinically credible. These numerical values are illustrative and should not be treated as recommended thresholds.
Applying the same delta to both treatment groups may not reflect the clinical context. Participants receiving control treatment might discontinue because of limited efficacy, while participants receiving active treatment might discontinue because of adverse events. Different assumptions may therefore be appropriate for each group.
A treatment-specific analysis might apply:
A bidirectional analysis can vary assumptions in both groups simultaneously. Results may be displayed as a heatmap in which one axis represents the active-treatment delta and the other represents the control-group delta. The display identifies combinations under which the study conclusion is retained or lost.
The numerical tipping point should always be interpreted together with its clinical plausibility.
A result is more credible when the conclusion changes only under assumptions that are substantially worse than clinicians consider credible. Conversely, if a modest and plausible departure from MAR overturns the result, uncertainty arising from the missing data may require closer consideration.
For example, if the tipping point requires missing active-treatment outcomes to be far worse than outcomes observed in clinically similar participants, the result may be easier to defend. If the conclusion changes after a small shift that is consistent with observed discontinuation reasons, the missing data uncertainty may be more material.
Clinical plausibility may be evaluated using:
The interpretation should also consider how many participants are affected. A large delta applied to a small number of missing values may have a different implication from a smaller delta applied to a large proportion of participants.
A sensitive result is not necessarily incorrect. Tipping point analysis describes the dependence of the conclusion on untestable assumptions, but does not establish which scenario is true.
The general objective is consistent across endpoint types, but the implementation must reflect how the outcome is measured and analysed.
For continuous endpoints, numerical adjustments can be applied directly to imputed outcomes.
Examples include:
The shift must be applied on an appropriate scale and in the correct direction. If imputation is performed on a transformed scale, the clinical interpretation of delta should be evaluated on that scale or after back-transformation.
Binary endpoints classify participants into categories such as responder or non-responder. A conventional numerical shift cannot be applied directly to an observed binary outcome.
Instead, investigators can vary assumptions about the probability or status of response among participants with missing outcomes. A shift applied on a log-odds scale may not correspond to a fixed change in response probability, so the chosen scale should match the question being tested. Approaches may include:
The tipping point is reached when the assumed response pattern among participants with missing data changes the predefined study conclusion.
For time-to-event endpoints, incomplete information often appears as censoring rather than a completely absent outcome.
Examples include:
Sensitivity analyses may examine departures from assumptions about independent or non-informative censoring. They may vary post-censoring event hazards, impute alternative event times, or evaluate assumptions about participants lost to follow-up.
There is no single tipping point method suitable for every time-to-event analysis. The method must reflect the endpoint definition, censoring process, estimand, and primary survival model. Methodological work on time-to-event tipping point analysis describes structured approaches for assessing robustness to departures from censoring assumptions.
Tipping point analysis is only useful if the scenarios being tested match the clinical and statistical question. Small choices around shift values, scale, missing-data patterns, and prespecification can change what the analysis is actually assessing.
A narrow range may fail to identify the tipping point. An extremely wide range may include scenarios with little clinical relevance.
The selected range and increments should be sufficiently broad to challenge the primary assumption while remaining interpretable.
A delta has different meanings on the original, transformed, standardised, and model-link scales.
The analysis should clearly describe the scale on which the adjustment is applied and explain how its magnitude relates to clinically meaningful differences.
Delta adjustments intended to represent alternative assumptions about missing outcomes should generally be applied only to imputed values. Changing observed values would address a different analytical question.
Missingness arising from administrative disruption may require different assumptions from missingness following adverse events, lack of efficacy, or disease progression. Where appropriate, assumptions can vary according to the timing or reason for missingness.
The sensitivity analysis must address the relevant source of uncertainty. For example, worsening only active-treatment outcomes may not be informative when the primary concern relates to missing control-group outcomes.
The tipping point analysis should remain aligned with the treatment effect defined by the estimand. Changing the population, endpoint definition, or handling of intercurrent events could result in an analysis addressing a different clinical question.
The SAP should describe the analysis method, affected participants, adjustment scale, treatment-group assumptions, scenario range, increments, pooling method, and criterion used to define the tipping point. Prespecification helps distinguish a planned robustness assessment from a post hoc analysis that may be harder to interpret objectively.
Tipping point analysis provides a transparent way to assess how strongly a clinical trial conclusion depends on assumptions about missing outcomes.
The tipping point itself is only the beginning of the interpretation. Its value depends on whether the scenario is clinically plausible, appropriately aligned with the estimand, and supported by a clearly prespecified analytical framework.
‘Tipping point theory’ is a broad term describing a threshold at which a relatively small additional change produces a different outcome or state. In clinical trial statistics, tipping point analysis has a more specific meaning. It identifies the assumptions about missing data under which a predefined study conclusion changes.
In clinical trials, ‘tipping analysis’ usually refers to tipping point analysis. It involves analysing a range of assumptions about missing outcomes and identifying when the study conclusion is no longer maintained. The method, assumptions, and decision criterion should be clearly defined because the shortened term does not describe the specific analytical approach used.
There is no universal acceptable percentage. The effect of missing data depends on the amount, reasons, timing, distribution between treatment groups, relationship with outcomes, endpoint, estimand, and analysis method. A small amount of systematically missing data can be important, while a larger amount arising for reasons unrelated to the outcome may have less effect.
The priority should be to prevent missing data where possible, document why data are missing, and assess the robustness of important conclusions using appropriate sensitivity analyses. The EMA guideline states that the aim should be complete data capture, including data from participants who discontinue treatment.
Quanticate’s statistical consultancy team supports sponsors with missing-data strategy, sensitivity analysis planning, and the statistical interpretation of clinical trial results. If you would like to discuss how tipping point analysis could fit into your study design, SAP, or regulatory submission strategy, request a consultation with our team.
Related Blogs