Quantitative graphical and synthesis approaches mentioned in the paper, with their main strengths and weaknesses in the context of complex interventions
| Methodological approach | Data requirements from each study | Main strengths | Main limitations |
| Forest plot (without overall effect) | Effect size and CI on the same metric | Widely familiar; each study clearly identified | Replication (of similar research questions) across studies is uncommon; effect size data may not be available |
| Albatross plot | P value, sample size and direction of effect | Data requirements are basic, so usually met; possibility of making indirect inferences on underlying effect sizes | Does not provide estimate of effect size; studies not clearly identified |
| Harvest plot | Conclusion of statistical test for effect; study feature(s) of interest | Data requirements are basic, so usually met; multiple outcomes can easily be displayed | Arbitrary distinction of studies according to statistical test; does not provide estimate of effect size |
| Effect direction plot | Conclusion of statistical test for effect; study feature(s) of interest | Data requirements are basic, so usually met; multiple outcomes can easily be displayed | Arbitrary distinction of studies according to statistical test; does not provide estimate of effect size; studies not clearly identified |
| Bubble plot | Conclusion of statistical analysis for effect; study feature(s) of interest | Data requirements are basic, so usually met; multiple outcomes can easily be displayed | Arbitrary distinction of studies according to result of statistical analysis; does not provide estimate of effect size; studies not clearly identified |
| Binomial test | Direction of effect | Data requirements are basic, so usually met | Does not provide estimate of effect size |
| Combining p values | P value and direction of effect | Data requirements are basic, so usually met | Does not provide estimate of effect size |
| Standard meta-analysis (eg, weighted average) | Effect size and CI (or equivalent) on the same metric | Widely familiar; produces effect sizes (important for decision making) | Replication (of similar research questions) across studies is uncommon; effect size data may not be available |
| Multiple outcomes meta-analysis (multivariate methods) | Effect size and CI (or equivalent) on the same metric for each outcome; data on correlations between outcomes | Can strengthen analysis of one outcome by 'borrowing strength’ from other outcomes | Requires reasonably large number of studies for reliable results |
| Subgroup analysis | Effect size and CI (or equivalent) on the same metric; study feature(s) of interest | Straightforward and widely familiar; flexible approach appropriate for examining impact of context, settings, participants, intervention characteristics | Addresses one study feature at a time; requires reasonably large number of studies for reliable results; high risk of false-positive conclusions; often has low power to detect true impacts of the features examined |
| Meta-regression | Effect size and CI (or equivalent) on the same metric; study feature(s) of interest | Allows multiple study features to be examined together; flexible approach appropriate for examining impact of context, settings, participants, intervention characteristics and for mediating effects of intermediate outcomes | Requires reasonably large number of studies for reliable results; high risk of false-positive conclusions; often has low power to detect true impacts of the features examined |
| Multiple interventions meta-analysis (network meta-analysis) | Effect size and CI (or equivalent) on the same metric; category to place each intervention | Facilitates rank ordering of interventions for the outcome | Requires interventions to be grouped into (reasonably homogenous) categories; requires similar target population for all studies; requires all categories of interventions to be 'connected’ in the network |
| Components-based approach to intervention complexity | Effect size and CI (or equivalent) on the same metric; components present in each intervention | Facilitates identification of most important component(s) of complex intervention | Requires reasonably large number of studies for reliable results; Assumptions required about whether components act additively or otherwise |
| Qualitative comparative analysis | Effect size estimates and study features of interest | Supports non-linear effects; multiple pathways to effectiveness; operates in ‘small n’ scenarios | Produces explanatory, rather than predictive, findings |
| Model-driven meta-analysis | Assumed causal model (logic model); effect size information for each relevant path in the model | Flexible approach to combining evidence; forces thinking about how effects arise | Dependent on appropriate assumptions being made in the causal model and availability of data |
| Mathematical models and system science methods | Assumed model; variable data requirements | Flexible approach to combining evidence; can supplement evidence with model-based assumptions when evidence is not available; wider focus beyond the intervention may include contextual information and dynamic interrelationships | Heavily reliant on assumptions going into the model; may require very large data sets |