Feeding the meta-analytical machine: piling the stack and singing in harmony

Thrombolysis was not routinely used to treat heart attack until the late 1980s. If the available trials had been combined in a meta-analysis, however, its effectiveness would have been established beyond reasonable doubt by 1973.6 Precedents like this suggest that the solution to a lack of evidence is simply to conduct more intervention studies in more places, on the basis that once we have a tall-enough stack of good-enough papers to populate a meta-analysis, we will know.

In practice, however, many systematic reviews of population health improvement strategies have been more successful in delineating what we do not know than in identifying unequivocally effective interventions.2 Often, what has prevented the formulation of clear answers is not so much a lack of studies as the lack of a way of reconciling the diversity of their study designs, limitations, interventions and contexts.7 One apparent solution is to limit meta-analysis to a set of more statistically comparable studies, but this risks perpetuating an evaluative bias in favour of an intervention ‘monoculture’ that may or may not include the most promising strategies.8 Another way of dodging the challenge is to split the problem into more and more discrete and evaluable chunks. These may eventually tell us the effect of doing X, but however refined the answers to this kind of ‘splitting’ question turn out to be, they are not sufficient to address the more pressing ‘lumping’ question for public health: how can we best achieve Y?9 10

If population-level intervention studies were to use a common set of exposure and outcome measures, this would make meta-analysis more feasible. Important progress has been made in this respect, for example in physical activity epidemiology.11 One might envisage some form of multicentre study in which more-or-less comparable interventions were introduced (or not) in different places and evaluated along the lines of a cluster randomised controlled trial. However, the Achilles heel of this vision is the qualifier ‘more-or-less comparable’. Some interventions, such as screening programmes, might be designed and implemented in a sufficiently similar way for this kind of multicentre evaluation.12 For upstream interventions in complex systems, however, the harmonised measurement of exposures (to interventions) and intermediate and final outcomes involving multiple causal pathways is challenging. Consider, for example, the array of measures of pricing, product formulation, purchasing, consumption, diet, health, and potentially confounding background trends that are needed to properly quantify the intended and unintended impacts of introducing a national levy on sugar-sweetened drinks.13 Negotiating the harmonised implementation of truly comparable interventions in multiple jurisdictions and beyond the control of researchers may be even less feasible.

Broadening the scope: building the panopticon and modelling the solutions

If empirical intervention studies are so difficult to design, implement or combine in meta-analysis, why not make more use of observational and simulation methods? The growth of ‘big data’ and interest in the ‘quantified self’ now offer unprecedented possibilities to gather enormous quantities of information, whether from surveillance systems such as traffic cameras or portable devices unobtrusively capturing continuous geographical, physiological and other signals from individuals. This torrent of data has led us towards a contemporary version of Bentham’s panopticon, telling observers exactly what people are or have been doing, where, when and even with whom. Datasets of this breadth, depth and precision make it possible to investigate associations with an unprecedented degree of statistical power and analytical complexity, and one might assume that so long as sufficiently rich data are available to populate such analyses, more and more secure causal inference will follow. The results can also be used as inputs to tools such as systems dynamic modelling—to simulate the consequences of altering upstream determinants of health, identify new intervention points and explore to what extent the outcomes observed in one system may be generalisable to others.14

However, enthusiasm for increasing computational complexity in the search for causal inference from observational or simulation data should be tempered with the recognition that design-based inference is generally considered stronger than model-based inference.15 In other words, we should attend at least as much to investigating situations in which different groups are exposed to different exogenous factors (interventions, or at least determinants of change) as we do to refining ways of eliciting ‘causal’ evidence from other datasets. Even if well-founded concerns about the representativeness and privacy implications of relying on ‘big data’ can be addressed, the resulting associational cornucopia is unlikely to help much if it contributes merely to producing ‘ever more sophisticated answers to the wrong questions’.16

For example, hundreds of studies now tell us that more walking is reported in areas where it is easier and safer for people to walk and there are places for them to walk to.17 However, precisely quantifying dose–response relationships of this kind does not necessarily explain how to address the problem of comparative inactivity, just as proving the aetiological case against tobacco did not explain how to reduce the prevalence of smoking.18 We should therefore not assume that the answers to the question of what we should do will be found by searching for statistical associations that only become noticeable in extremely large samples. Cohort and surveillance data collected for other purposes can certainly be used to investigate the effects of interventions,19 but no matter how intensively people’s health, behaviour and environments are quantified in observational studies, it may be a category error to assume that this will necessarily explain whether or how public health strategies actually work (or not). Epidemiology is only one of the tools in the box.20

Looking beyond ‘interventions’

For most public health interventions—even well-established population screening programmes—the only honest answer to the question ‘Does it work?’ is ‘It depends’.14 22 In most cases, the questioner will need to clarify what they mean by work (in what terms?), what they mean by it (what, exactly, is the intervention?) and indeed what they mean by does (which implies a generalisable inference). Sometimes, what people really mean is ‘Will it work?’—a predictive question—or ‘Did it work?’—an empirical but also a particular question, perhaps better formulated as ‘What happened?’10

Why is this so difficult? Most public health interventions are at least somewhat unique to their context, which ought to be taken account of in their evaluation, and many can also be seen as interventions in complex systems.10 12 22 However, they do not necessarily take place in a context, at least not in the sense that a new clinical procedure might be introduced in a certain hospital or healthcare system. Rather than exerting an effect within a ‘moderating’ context, it may be more helpful to see these interventions as targeting and altering the context in which people live and make choices.10 23 24

But if everything is complex and context ‘is’ the intervention, what exactly might we seek to generalise from one instance to another? We tend to assume that interventions are things that work in and of themselves and might be universally generalisable, like Newton’s laws of motion.25 26 In practice, however, we find ourselves struggling to make sense of an apparently incommensurable body of evidence. By adopting a ‘naive and misplaced commitment to the reproducibility of the complex’, we may have unwittingly set ourselves an unrealistic challenge of identifying generalisable interventions as such.22 27 28 What if we were to release ‘our search for universal generalisability in favour of more modest, more contingent, claims’?22 Among other things, this would entail relaxing our grip on the notion of generalising the effects of interventions based on their forms or their ‘active components’, and turning our attention instead to their functions—the processes and changes they evoke—or indeed their ‘spirit’.12 26

To take a well-known clinical example, some reviews have found that the type of psychotherapy offered to a patient with depression makes little difference to the outcome.29 What to do, then? Others, taking a different approach to analysis, have found that the key lies in the quality of the therapeutic relationship established rather than the particular techniques used.30 Reasoning by analogy, we theorised that in another arena—promoting active travel—the myriad forms of intervention might be underlain by a more limited number of critical functions, such as increasing accessibility or safety, that are generalisable in principle but might be achieved in different ways in different situations.31 Again, this is as much to do with practical strategies as it is to do with theories, and some public health guidance and government policy already implicitly reflects this way of thinking with references to high-level principles, variable interpretations and the like (examples: table 1).

Searching for patterns

If this idea has traction, we will need to expand the scope and flexibility of our repertoire in evidence synthesis in order to derive more concrete and defensible inferences about what to do for public health. Rather than assessing whether interventions of a particular type ‘work’ in an overall sense, this will entail aggregating evidence for and against theories about intervention functions by combining information from studies conducted in different situations, including studies that were not explicitly designed with this in mind.7 14 27 32 33

How might we do this? We could accept the limitations of relying so heavily on testing binary statistical hypotheses about singular study outcomes,2 14 and turn our attention to seeking ‘inference to the best explanation’—that which provides the greatest understanding.18 34 35 We could use intervention theory to predict patterns that might be observed in a variety of data, and then assess the concordance between the observed patterns and the theoretical expectation patterns—testing theories rather than interventions.33 36 We could go further by systematically considering alternative potential explanations for the patterns we observe, doing our best to confirm or disconfirm these, and reaching a conclusion as to the most plausible causal inference from the overall pattern of findings. The approach is most easily illustrated within a single study, an evaluation of new transport infrastructure that was not designed to test any singular overarching hypothesis (case study: table 2).

One might counter that the principle of comparing observed and expected data applies equally to the paradigm of the randomised controlled trial. While this is true, testing theories in the way we describe entails a more radical challenge to established notions of a hierarchy of study design. For example, it is often understood that quantitative methods are for testing hypotheses, whereas qualitative (and some quantitative) methods play more subservient roles such as generating hypotheses, developing interventions or assessing their acceptability.32 One might also counter that approaches such as process evaluation, or realist evaluation and synthesis, already offer ways of investigating causal mechanisms.7 24 27 37 While this is true, the higher-order intervention functions and data patterns we are talking about are likely to reflect multiple underlying mechanisms25 and others have argued that more diverse lines of evidence should be converged and brought to bear on the challenge of overall causal inference. These might combine a variety of quantitative sources of causal estimation with a variety of quantitative and qualitative sources of causal explanation such as causal process observations.15 22 34 Simple examples from historical and contemporary communicable disease control illustrate this principle (table 3).

Embracing the mess

Some public health strategies will inevitably be more successful than others, and every ‘solution’ has the potential to generate more problems. Such uncertainty is—or at least should be—what drives scientific enquiry in the first place.38 Rather than hoping for 'a neat, coherent story’ of clear-cut outcomes from evaluation, therefore, in most cases we should expect confusing, divergent, mixed or unexpected patterns of results.21 Far from denoting that an intervention or evaluation has failed, these shed light on what really happened, whether we like it or not.10

While the metaphor of the dry stone wall can be applied at the level of the individual study, as shown in table 2, the mess of this apparent dissonance may be better resolved not at the level of the individual study, but by cumulating evidence from multiple studies over time within an intervention research programme or a systematic review. Our final case study illustrates how we applied the principles of linking diverse sources of evidence on causal estimation and causal explanation to identifying patterns and testing theories about intervention functions across a cumulated body of work on infrastructure to support active travel (table 4).31 It draws on a variety of research methods, resting on different philosophical assumptions, in pursuit of ‘clarification and insight, for which a more interpretive and discursive synthesis is needed’.3 9