False positive results

The probability of a false positive result when testing the effect on one specific outcome is typically measured using the p-value from the statistical test. A p value of 0.1 implies that there is 1/10 chance that we identify an intervention effect when there was no true effect. If all trials fully reported all outcomes tested, then the overall probability of false positive results among all estimated effects would be accurately measured by the aggregate p value, but frequently trial reporting is done selectively. Selective reporting of outcomes on which the intervention had a positive effect (and under-reporting of null findings) is well documented and contributes to publication bias.1 7 21 22 Null results (ie, the absence of an observed effect), or any observed effects opposite to the desirable one, are much less likely to be written up and published. Not reporting of unfavourable results might be due to possible economic gains (eg, selling a solution to a problem) and academic incentives (eg, it is easier to get a paper published when positive large effects are reported).23 When trial reporting is done selectively, based on the study findings and not simply on the evaluation questions the trial sought to answer, our collective understanding of what works is biased. To illustrate the problem, assume that the same intervention is tested in 40 different well conducted and adequately powered trials and that 1 of these trials documents an effect on the outcome of interest. If all trials are fully reported, then the evidence base will show that the effect was observed in only 2.5% of the trials, likely reflecting a false positive result. If only the trial that found a positive effect is reported, however, readers will wrongly conclude from the evidence that the intervention is effective for that outcome.

False negative results

The ability to find a positive effect on an outcome when it truly occurred depends on the power of the study. Statistical power is one minus the probability of a type II error. A statistical power of 80%, which is frequently used in efficacy and effectiveness trials, leaves a 1/5 chance that an actual effect is not detected. Increasing power to 90% reduces that probability to 1/10. Underpowering a study leaves ambiguous the interpretation of a null result (ie, not finding an effect) because true absence of an effect cannot be separated from the effect being too small to be detected given the limited power.

Underpowering studies also negatively influences how studies contribute to the evidence base. Since underpowered studies can only detect large effects, small effects often remain unreported. Selective reporting of the large effects will bias the average effect in the published literature upward, that is, the average magnitude of the impact documented in papers will be much larger than the true expected effect of the intervention.

Inadequacy of typical recommendations for trials of complex interventions

To safeguard the accuracy of inference from RCTs, regulatory agencies (eg, US Food and Drug Administration, the European Medicines Agency), research funders (eg, UK Medical Research Council) and groups of trial experts (eg, Consolidated Standards of Reporting Trials or CONSORT) have devised guidelines on how to design, implement and report on RCTs.7 13 14 24–30 A key feature of these guidelines, which target trialists in health-related fields, is to declare primary and secondary outcomes (table 1). The distinction between primary and secondary outcomes is uncommon in RCTs conducted in other fields such as economics.1 31 32 Another common feature across these guidelines targeted to health researchers is to limit the number of primary outcomes to a few; some guidance states that the number should be one. Guidelines generally agree that primary outcomes relate to the primary study objective, that they need to be prespecified, and that they drive sample size requirements to ensure adequate statistical power. Secondary outcomes are inconsistently defined.

The recommendation to limit the number of outcomes, especially primary outcomes, stems from concern about a higher probability of false positive findings with a larger number of outcomes, which is transformed into concern about multiplicity. We discuss below that multiplicity is typically not a threat to the accuracy of inference from RCTs of complex interventions. The appropriate strategy to help minimise the problem of false positive findings is to declare and register outcomes in advance, use one measure per outcome and report on all outcomes irrespective of whether an intervention effect was found. Excessive limiting of the number of outcomes comes at an unacceptable cost as it restricts what can be learnt from the evaluation of a complex intervention by unnecessarily forcing researchers to not assess the intervention’s effect on outcomes important to decision makers and on outcomes along the impact paths.

The guidance to prespecify outcomes implies that outcomes are declared and registered in a public trial registry before the trial is started. Registration ensures transparency about the preplanned outcome(s) purported to represent whether the intervention worked as intended and helps to address the problems of false positive findings and selective reporting. The recommendation to adequately power all primary outcomes reduces the probability of false negative findings.

The formal guidelines have been supplemented with publications in the scientific literature that describe how the type of outcome determines the type of inference.33 34 Based on typical recommendations, common thinking among many trialists, including many complex intervention trialists, can be summarised as (1) only primary outcomes allow the researcher to draw confirmatory (ie, causal) conclusions and (2) conclusions based on the secondary outcomes are exploratory and not confirmatory. This belief, however, is not supported by the existing guidelines and does not logically follow from the attributes of primary and secondary outcomes as described in these guidelines. The strength of the causal inference for any specific outcome depends on the study design and implementation and not on the label given to the outcome. The strength of a well-designed and properly implemented RCT lies in the ability to conclude that the observed effect on an outcome is due to the intervention and not to something else. The ability to reach this causal conclusion is the same for each trial outcome, irrespective of the number of outcomes, the label the researcher assigned to the outcome (ie, primary or secondary), and whether the outcome was declared before the trial or after the trial was concluded. What matters is the accuracy of the inference that can be made from the perspective of the larger evidence base. That is, what is the probability that some of the observed effects are false positive findings, are the effects estimated with confidence only the ones that are exceptionally large due to limited statistical power, and what is the probability that not finding an effect reflects the true absence of an effect? Whether causal inference is confirmatory or exploratory can only be determined relative to the larger evidence base and not with respect to any individual outcome. That is, confirmatory and exploratory are properties of the inference and not of specific outcomes.

In summary, current guidance and recommendations address many but not all the concerns related to false positive and false negative findings. Simply classifying outcomes as primary and secondary does not sufficiently protect against these two types of false findings. Limiting the number of outcomes to just a few unnecessarily restricts what can be learnt from RCTs of complex interventions. The number of outcomes is discussed later in the manuscript.

Multiple outcomes of interest

When a set of hypotheses (each about a different outcome) is tested simultaneously, the overall type I error rate, that is, the probability of wrongly rejecting at least one null hypothesis, increases.40 Simultaneous hypothesis testing assumes a universal null hypothesis that the intervention has no effect for all the outcomes investigated versus the alternative hypothesis of impact on at least one of these outcomes. Adjusting for multiplicity is only warranted if a set of hypotheses is tested simultaneously in this formal sense,24 40 but this test of a universal null vs the stated alternative hypothesis is nearly always irrelevant to the scientific or evaluation questions being investigated.40 42 When each hypothesis is limited to a single outcome, as is typically the case in impact evaluation, the probability of a false positive result remains the same irrespective of whether one or a million comparisons are tested.40 Because the p value is a simple transformation of a test statistic (eg, t-statistic or χ² statistic) for a given outcome to a 0–1 scale using the assumed (eg, t or χ²) distribution, the type I error rate holds for that outcome regardless of whether other outcomes have been similarly transformed.

We illustrate this with the example of the Nutrition Embedded Evaluation Programme Impact Evaluation (NEEP-IE) trial in Malawi, which evaluated the impact of a complex intervention.43 The intervention was designed to improve child nutrition through several paths. First, the intervention was expected to increase agriculture production which would subsequently improve household-level availability of nutritious foods. Second, the nutrition behavioural change communication sought to improve diets and feeding practices by increasing caregiver knowledge. Third, by increasing the regularity and quality of meals in the community-based childcare centres (CBCC), the intervention was expected to promote CBCC participation and enhance learning and nutritional status.

The NEEP-IE trial had several main outcomes and corresponding hypotheses (online supplemental table 1). Multiplicity correction would have been required if the authors had tested the universal null hypothesis that NEEP-IE did not have an impact on any of these outcomes versus the alternative hypothesis that it had an impact on at least one outcome, that is, ‘the intervention increased agricultural production OR increased the diversity of agricultural production OR increased preschooler CBCC enrollment OR increased preschooler CBCC attendance OR increased child dietary intake’. This simultaneous hypothesis testing only makes sense if the researchers defined programme effectiveness as at least one of these five outcomes having changed meaningfully. This is, however, not how researchers generally draw conclusions about the effectiveness of complex interventions. The expectation that distinct, independent and perhaps alternative paths of impact are operating negates the utility of the universal null hypothesis. Even within a specific impact path, however, simultaneous hypothesis testing is not helpful to assess the effectiveness of the intervention. Instead, the impact of the intervention is first assessed for each of the individual study outcomes. The impact across all outcomes is then used to understand how and why the intervention improved some outcomes but not others. In the NEEP-IE study, the authors concluded that the intervention increased household production and diversity of production and had positive effects on sibling dietary diversity but did not increase CBCC enrollment or attendance. The authors did not make an overall (ie, intervention-wide) effectiveness statement. Instead, they used the study findings to identify why the programme was effective at improving some outcomes and not others.

Unnecessarily adjusting for multiplicity is costly. One direct consequence of decreasing type I errors (which is the objective of multiplicity adjustments) is that type II errors (ie, not identifying an impact when it exists) increase.24 40 42 The only solution is to increase the sample size, which has a financial and a societal (eg, economic, effort, time) cost, since more study participants are needed. Using more resources and enrolling more study participants than needed is unethical.44 Another direct consequence is that inferences may be inaccurate because in many situations outcomes are not independent as some multiplicity adjustment procedures assume,40 42 resulting in conservative, that is, biased tests.

Multiplicity concerns have led some to recommend combining several outcomes into a composite outcome measure,13 14 26 but composite outcome measures create interpretational challenges and often lack real-world relevancy.40 45 For example, the composite indicator of minimally acceptable diet for infants and young children is less useful for decision-makers, because of the difficulty to meaningfully interpret and act on it, than are the component indicators of minimal dietary diversity and minimal meal frequency.46 47 A positive effect on a composite outcome measure provides no information on which of the individual outcomes were affected. Likewise, a null effect on the composite outcome measure may hide a meaningful effect on one or several of the individual outcomes. Since complex intervention trials typically seek to understand how the intervention affected multiple outcomes, the use of composite outcomes measures is not recommended.

In summary, post hoc analytical adjustment for multiplicity resulting from multiple outcomes should not be done unless simultaneous hypothesis testing was planned, which is unlikely in a complex intervention trial.24 40 42 Careful selection of outcomes based on programme theory and having one measure per outcome (and not making a composite of measures) helps prevent an excess number of hypothesis tests.

Subgroup analyses

Subgroup analyses often are undertaken to understand differences in effects across groups defined by baseline participant characteristics such as child age or household wealth.41 If no hypotheses about subgroups were planned, then the analyses are exploratory and should be identified as such with no need to adjust for multiplicity. If the subgroup analyses were planned, that is, hypotheses have been stated, then adjusting for multiplicity is not necessary if all planned subgroup analyses are reported and the interpretation of them is done carefully. Rather than adjusting for multiplicity, the investigator and readers should examine all p values or CIs of the set of subgroups analysed and assess the likelihood that the findings would be due to chance considering the number of subgroups analysed. An important concern is that subgroup analyses will often be inadequately powered and thus more likely to produce false-negative results, providing an additional justification why they should be considered exploratory. Furthermore, randomisation typically is not conducted by subgroup which means that the arms may not be equivalent for all participant characteristics at the beginning of the trial especially when subgroups are small. Within subgroups, differences in outcomes across arms may thus be due to pre-existing differences between participants. In sum, only when subgroup analyses are adequately powered and prespecified should the findings be considered confirmatory causal evidence.

More than two study arms

Trials often have more than two study arms. Early in the last century, statisticians understood that doing a separate trial for each intervention was wasteful, inefficient and prevented investigating interactions.48 For example, the MINIMat RCT in Bangladesh had 12 arms formed by the combination of early vs usual food supplementation, three types of micronutrient supplements, and intensive versus usual lactation counselling in a factorial design.49 This design allowed the examination of all three types of interventions at the same time and the testing of the principal hypothesis which was that the synergistic combination of early food supplementation and multiple micronutrients would be superior to the other five combinations of nutrition interventions during pregnancy. Standard methods for optimally handling multiple comparisons arising from multiple study arms have been developed. The principles for these methods are to (1) plan the comparisons in advance, (2) keep the number of comparisons for a given outcome to one less than the total number of arms48 and (3) if these first two conditions do not hold then use a procedure optimised for the type of comparison (eg, Tukey test for all pairwise comparisons).48

Initial comparisons prompt other comparisons

In any scientific study, findings from initial comparisons may prompt further examination of other comparisons or relationships. Whereas the initial comparisons ideally are preplanned, the comparisons that follow are not preplanned and must therefore be considered exploratory. For transparency, investigators should disclose which comparisons were preplanned and which were not, and how the decision to pursue further comparisons depended on the initial findings.

Analyses repeated at multiple visits

When participants are assessed at multiple visits, analyses could be repeated with data from each of these visits. In trials of complex interventions, data from multiple visits are typically used to understand the unfolding of effects over time; analyses can also be done with all visits together. Investigators should decide and declare in advance in the trial protocol or analysis plan how the analyses will be conducted for each outcome. In some cases, the multiple measurements may be reduced to a single effect estimate such as the difference between the first and last visits or the linear trend over all visits. When multiple visits are to be used to estimate an overall effect, statistical procedures that appropriately account for the lack of independence among the visits must be employed.

Interim analyses

Interim analyses are usually done to consider early stopping of trials to address ethical concerns about potential harms or when evidence of effectiveness is sufficient to not deprive participants in a comparison arm of demonstrated benefits. Accepted methods for conducting interim analyses have been established.41 Interim analyses are not usually done in trials of complex interventions.

Assessment of implementation of the intervention

For an intervention to be effective, the intervention components must be implemented with fidelity, reach participants, be delivered to participants, and be received by participants.50–52 Details about the experience, training and monitoring of implementers and the timing, intensity, fidelity, reach and dose of the interventions should be assessed through process evaluation and implementation research and reported. Measuring these implementation-related characteristics of interventions with multiple components and programme impact paths can be challenging. For example, an intervention may involve agricultural training, behaviour change communication, visits to health clinics, management of microcredit loans, and other activities. Specifying what is the adequate fidelity, reach, and dose across intervention components a priori in an intervention protocol is important to be able to understand the effectiveness of the intervention and to conduct analyses that attempt to account for less than full adherence to the intervention (see below).

Selection bias and attrition

Selection bias arises when inherent differences between the individuals (or groups) assigned to trial arms result in biased effect estimates. Selection bias can arise at any stage of a field-based study from the initial recruitment through to completion of the data collection. Random assignment minimises (but does not eliminate) the potential for selection bias in the early stages of a study by making the trial arms comparable. Attrition bias may result from loss of participants during a study, which may be particularly problematic in trials of complex interventions, if attrition is large and/or differential between arms.

In trials of complex interventions, selection and attrition biases can be large and can result from multiple forces because people in the intervention arm are subject to the intervention protocol and people in the comparison arm are not. Assignment in complex intervention trials should be concealed before consent to participate is given whenever possible. Given the nature of complex interventions, blinding is typically not possible because individuals living in a village, for example, can become aware of the presence of the intervention. Agreement to participate in the trial should be obtained before random assignment at the cluster level; the intervention, however, may have started before individuals consent to being in the study. A cluster-randomised trial evaluating the impact on pregnancy outcomes, for instance, may enrol a new cohort of pregnant women each month. Participation in the data collection and adherence to the intervention, although theoretically distinct, can be intertwined. For example, participants in the intervention arm who agree to participate in the study or who adhere more to the intervention protocol can both be more motivated, or more in need, than those who do not agree or adhere less. Those more motivated or in need may be more likely to have data collected and/or to stay in the data collection. On the other hand, people who are in the intervention arm may find the intervention protocol burdensome and be more likely to drop out of the study.

Minimising selection and attrition and documenting the selection and retention processes are important. If possible, minimal data on participants who decline participation during the recruitment and enrolment stages should be collected for comparison with study participants. After assignment and throughout the follow-up and data collection periods, procedures should be the same in all arms to the extent possible. Although participants assigned to a complex intervention may engage with only some intervention components or only some of the time, they should be followed in the data collection regardless of receipt of or engagement with the intervention.53 To help assess the consequences of attrition, minimal data collection with those who drop out may be possible, even if full data collection cannot occur for those otherwise lost to attrition.54

Decision on the type of analysis

Although the validity of the counterfactual is determined by the study design and its execution, the type of analysis that is used affects how well the trial provides a valid counterfactual for the intervention arm. The primary types of analysis are intent-to-treat and per-protocol, with protocol referring to the intervention and not to the data collection or analysis

The intention-to-treat analysis estimates the average impact of the offer of the intervention on outcomes, rather than the impact of participation in the intervention. Since many study participants may not fully adhere, or adhere at all, to the intervention protocol, the estimates of effectiveness may be lower than the efficacy of the intervention, that is, what effects would have been achieved if the fidelity, reach, dose delivered, and dose received were ideal. Intent-to-treat means that all eligible participants regardless of adherence with the intervention protocol should be included in the analysis of results whenever possible, in contrast to per-protocol which is an analysis of only those who adhered to or fully took up the intervention protocol.55 Collecting data on all who were included initially in all arms allows for comparison of those who were assigned to the intervention activities (regardless of fidelity, reach, dose delivered to them or dose received by them) with those who did not. Comparison of completers and drop-outs from the data collection in all arms helps to assess attrition bias. In practice, fully achieving an intent-to-treat analysis is often difficult because of missing data resulting from attrition if the study follows individuals. The intent-to-treat analysis is thus conducted on participants for which the outcome is known. Attrition is documented in each arm.

A per-protocol (ie, as-treated or treatment-on-treated) analysis restricts the analytical sample in the intervention arm to those who fully adhered to the intervention protocol, which needs to be specified prior to the analysis.56 Measuring adherence in trials of complex interventions, however, is difficult because of differences in adherence to the multiple components of the intervention.

Since study participants who fully adhered to the intervention protocol may have been more motivated or able to adhere, the estimates of effectiveness may be biased because those fully adherent are self-selected. Sometimes per-protocol analyses are modified to include those with partial adherence to try to balance the possible underestimation of the potential intervention effects that comes from an intent-to-treat analysis when there is less than full adherence and the bias that comes from restricting the analytic sample to only those fully adherent to the intervention protocol.57 An alternative method to account for less than full adherence is the use of instrumental variables to estimate the local average treatment effect, that is, the effect of the intervention for those who adhered to it.1 58 59 Another type of analysis is the dose–response analysis, where participants are classified based on the level of adherence or exposure to the intervention. This analysis is used to understand if observed effects are associated with participant adherence. It has the same associated biases described above.

The CONSORT statement indicates that intention-to-treat is the preferred analysis type and acknowledges that performing strict intent-to-treat analysis is challenging.7 An intention-to-treat analysis should always be reported for at least three reasons. First, it takes full advantage of the random assignment to establish a valid counterfactual. Second, if the intention-to-treat results show that the intervention had effects despite having people in the analytic sample who did not fully adhere to the intervention protocol, then the results provide strong evidence of effectiveness even if it is an underestimate of the intervention’s (biological or behavioural) efficacy. Third, the intention-to-treat estimates reflect what policy makers, funders, and programme implementers can reasonably expect the average population impact to be when the programme is implemented at scale. The estimates of intention-to-treat effects thus may be more relevant for policy than estimates of efficacy. A per-protocol or instrumental variable analysis may provide useful information in addition to the intention-to-treat analysis but should be considered as a secondary analysis designed to help clarify why, or why not, an intervention had its observed effect.

Step 1: specify the programme theory

The programme theory (also referred to as theoretical model, theory of change, causal paths or paths to impact), which identifies the key intervention components and how they are expected to be linked to achieve impact, is designed by the group of programme implementers, evaluators and other relevant stakeholders. Displaying the programme theory in a programme impact path diagram is an essential exercise for recording the programme theory.60 61 The programme impact path diagram should specify both the intervention actions and the changes in the targeted participants that are together expected to generate benefits.62

Step 2: identify outcomes and measures

Using the programme impact paths, all outcomes that the intervention is designed or expected to affect are identified. The list of outcomes should include those that represent the end and intermediate points along the impact paths, both desirable and undesirable, and should include both what implementers and participants do (ie, actions they are expected to take) and what changes are meant to occur. Primary outcomes (ie, relevant based on the intervention intent and programme theory) should be identified. These outcomes will be accompanied by hypotheses and be adequately powered (step 3). Trials of complex interventions are likely to have many primary outcomes.

Once all outcomes have been identified, measures that reflect these outcomes need to be chosen. The choice of outcome measures is determined by validity, measurement feasibility and cost. Ideally one measure per outcome is chosen. Sometimes multiple questionnaire items are intended to be combined to form a scale to measure an outcome.

Step 3: calculate sample size requirements for adequate power

Sample size calculations determine the number of observations needed to detect the minimum meaningful effect of the intervention on each of the identified primary outcome measures. For trials that cannot be easily repeated such as those typically conducted to evaluate complex interventions 80% power which is frequently used in trials may be too low, and we recommend that at least 90% power is used. The outcome with the highest sample size requirement will determine the overall study sample size. Finally, additional sample size provisions need to be made for planned subgroup analyses and planned tests of effect modification as well as attrition, missing data and any other problem that will reduce the number of observations that can be included in the data analysis.38 Sample size requirements for some outcomes may be beyond the study’s budget. These secondary outcomes must be identified as inadequately powered; although they can be assessed, inference will be only exploratory (table 2).

Step 4: register the trial and produce the statistical analysis plan

In the trial registration (online supplemental table 2), which should be completed before the trial is started, investigators declare the outcomes that they will analyse, and which outcomes are adequately powered. The statistical analysis plan should provide details about (1) administration of the study; (2) background, rationale and objectives; (3) study methods; (4) statistical principles; (5) trial population and (6) planned analytical methods.63 The statistical analysis plan is an agreement among the principal investigators, possibly other investigators, stakeholders or sponsors, the senior statistician and data analysts. It should be completed and registered before analysis begins. It should be detailed enough for other analysts to use the data, conduct the analysis, and arrive at the same results. Any deviations from the analysis plan should be documented. The study protocol, statistical analysis plan, and trial reports and publications need to distinguish between primary outcomes, that is, those that are declared (ie, registered) and adequately powered, and all other outcomes (table 2).

Step 5: assess programme implementation

All activities to standardise the interventions across participants should be described. Variation in the timing, intensity, fidelity, reach and dose of the delivered interventions should be fully documented through a thoroughly designed process evaluation.

Step 6: focus on declared and adequately powered outcomes and limit the inference for undeclared and inadequately powered outcomes

Confirmatory causal inference is limited to the primary outcomes, that is, the outcomes that were declared and registered before the trial was started and for which the study was adequately powered (table 2). Inference about outcomes that were not declared before the trial started should be considered exploratory because having clarity on how and when decisions were made to test these outcomes is difficult. The interpretation of the absence of an effect depends on statistical power. When no meaningful and feasible effect is observed and the trial was adequately powered, we can conclude (with a quantifiable level of confidence) that the trial did not have an effect of the magnitude for which the trial was powered for that outcome. When a trial was not powered to detect a feasible and meaningful effect, however, a finding of no effect could be because either the effect is truly absent, or the effect was too small to be detected. In this case, researcher should avoid interpreting these results as the programme not having an effect.

Step 7: maintain transparency when reporting outcomes

The disposition of participants should be clearly documented through a CONSORT flow diagram.7 If the trial used cluster randomization, the diagram should report on the disposition of the clusters and individual participants by arm, including consent and enrolment rates.

All planned outcomes should be reported irrespective of whether an intervention effect was found, and all unplanned outcomes and analyses need to be identified. For each outcome, an estimate of effect (even if small or null) and the confidence in that estimate (ie, exact p value or CI) are reported without adjustment for multiplicity of outcomes so that readers can interpret what was found for each outcome.

Even when outcomes are declared and registered before the trial, researchers often only report on those outcomes that the intervention affected. Although reporting all primary outcomes in one paper could eliminate this problem of reporting bias, trials may have more primary outcomes than can be accommodated in a single journal manuscript. Furthermore, collaborating researchers may want to report the impact on specific outcomes in different disciplinary journals. We recommend the mandatory inclusion in each trial paper of an overview of all primary and secondary outcomes as declared in the trial registration, the outcomes analysed and reported in the paper, references to papers for previously reported outcomes, and outcomes that remain to be analysed and reported. The proposed recommendation will increase transparency and provide a direct quantifiable measure of potential publication bias, that is, the extent to which researchers report on some (but not all) of the registered outcomes. Journal editors and reviewers are important for ensuring transparency by thoroughly reviewing submitted trial manuscripts. Their review should include carefully comparing the reported outcomes against the outcomes that were registered, assessing whether outcomes were powered and checking whether the authors followed the statistical analysis plan.