• cluster randomized trial
• child health
• public health
• mathematical modelling

Colbourn et al 1 raise three interesting questions. (1) Whether, if a randomised controlled trial (RCT) fails to detect an effect on the primary outcome but shows a significant effect on an intermediate outcome, it is legitimate to model the effect? (2) Whether the modelling was performed correctly? and (3) Whether it is meaningful to focus on demand-side factors in isolation?

(1) We would argue that it is legitimate to use a model to evaluate an effect that the RCT was not, and could not be, powered to detect, where there is strong evidence of an impact on an important intermediate outcome, provided the limitations of such an exercise are appropriately acknowledged.

Measuring the effect of media campaigns is challenging. Very few countries have the very low national radio and television penetration and high local radio listenership that make it possible to evaluate a radio campaign through a cluster randomised trial.2 Even in those few countries where media configurations are suitable, the power to detect important changes in mortality is constrained by the mortality rate and the number of clusters that can be achieved (14 in the case of Burkina Faso).

When we began our study in 2011, we designed our trial to have 80% power to detect a mortality reduction of 20% but just 30% and 8% power to detect reductions of 10% and 5% respectively. Due to the constraints outlined above, it was impossible to power the study to detect small reductions. Rapidly falling mortality during the study period (from 115.5/1000 to 76.5/1000 in the control arm) will have further reduced our study’s power. Given the lack of power to detect a reduction in mortality in the range of 5%–10%, it is important to remember that ‘absence of evidence is not evidence of absence’. Colbourn et al attach considerable importance to the point estimate, but our mortality estimates do not preclude anything from an important negative effect to an important positive effect.

More generally, it is also important not to overstate the certainty provided by survey-based methods which rely on mothers’ recall to estimate child mortality trends in countries lacking fully functioning vital registration systems. Mortality surveys can be subject to uncertainties greater than the effect sizes we were attempting to measure.

We did observe important changes in behaviour, using health centre records of over 600 000 consultations: a 56% increase in parents taking their children for malaria treatment (p<0.001) in the first year and similar results for pneumonia (39%, p<0.001) and diarrhoea (73%, p<0.001).3 Given the survey’s lack of power to detect smaller changes in mortality, we used LiST to estimate the effect of increased care seeking on mortality. LiST is a widely used and accepted tool for estimating the likely mortality impact of different interventions. The cost-effectiveness calculations simply follow from the mortality reductions predicted by LiST. The stated conclusion of our study was that ‘large-scale mass media campaigns encouraging prompt care seeking for children can be highly cost-effective in improving child survival.’4

(2) Colbourn et al raise four methodological concerns:

1. With regard to the ‘compression’ of the consultation counts, we adopted this approach in the context of multiple diagnoses during one consultation to avoid double-counting deaths averted. The method was applied consistently and correctly to all three diagnoses.

2. For pneumonia treatment rates, we used the LiST default approach which is to use the consultation rate with no adjustment for treatment rate. LiST takes this approach based on evidence that pneumonia treatment rates estimated from survey data are not valid.5 6 This has limited effect on the projections because the assumption applies to both the intervention and control groups (and because pneumonia represents a small percentage of the total lives saved). If we reduce the percentage of pneumonia cases treated from 52.7% to 44.8%, corresponding to reduced treatment arising from amoxicillin and cotrimoxazole stockouts, 7 the effect on the projected lives saved is small (from 7.1% to 6.9%).

3. The reasons for the relative decrease in consultations for ‘other diagnoses’ in the trial’s intervention arm are unclear; however, it is worth noting that ‘other’ diagnoses actually increase in both arms. This could possibly be due to strengthening in the implementation of Integrated Management of Childhood Illness. However, because the ‘other’ consultation data were aggregated, we lacked the data to investigate which diagnoses contributed to these trends.

4. Colbourn et al are correct that the lower bound of the reported lives saved for 2014, reported in table 1 in Murray et al 3 should be negative. The correct lower bounds for 2013 and 2014 should be 35 and −14, respectively, with the overall lower bound being 798 and not 1110. This error affects only the lower bounds and not the overall percentage reduction in mortality which remains at 7.1%.

(3) While Colbourn et al are, of course, correct that health systems are complex and that treatment seeking is worthless without available treatments and a functioning health system, the reverse argument is also true: if sick children are not brought for treatment, no health system is going to be able to save their lives. Very few would argue that appropriate treatment with effective antimalarials, antibiotics and oral rehydration salt (ORS) cannot save lives, and WHO and UNICEF have recommended prompt care seeking for these conditions for many years. 8 9 Care seeking is an important intermediate outcome and a precondition for everything that follows in the causal pathway leading to deaths averted. There are therefore strong grounds for believing that, if the health system is functioning reasonably well, improvements in care seeking can save lives.

In Burkina Faso, the evidence suggests that the health system was functioning reasonably well. As we note in the article, the WHO 2014 Service and Availability Readiness Assessment report indicates good availability of key life-saving treatments in primary health facilities in Burkina Faso: 91% for malaria treatment, 82% for ORS sachets and 83%–85% for antibiotics.7

In conclusion, we consider that it is legitimate to use a well-established modelling tool to estimate a health impact when there is strong evidence of an impact on an important intermediate outcome but little or no power to detect an impact on the health outcome itself. The limitations of such an exercise, of course, must be appropriately acknowledged.

However, we are in complete agreement that demand-side interventions are not a substitute for health systems strengthening or other supply-side interventions. Our study simply shows that demand-side interventions should not be confined to the periphery of public health and have the potential to save lives when used as part of a functioning health system.