Methods to assess the costs and health effects of interventions for improving health in developing countries

Introduction

Assessment of the cost effectiveness of interventions designed to achieve the millennium development goals for health is complex. The methods must be capable of showing the efficiency with which current and possible new resources are used, and incorporating interactions between concurrent interventions and the effect of expanding coverage on unit costs.1 They should also allow valid comparisons across a wide range of interventions. Here we describe how the standardised cost effectiveness methods used in the World Health Organization's Choosing Interventions that are Cost Effective (CHOICE) project have tackled these issues.

Level of analysis

The analysis was performed for 14 regions classified by WHO according to their epidemiological grouping (table A on bmj.com). The regional results (except if not relevant to the disease area, for example, malaria) are available at www.who.int/choice, but the papers in this series give details for just two regions: Afr-E, which includes countries in sub-Saharan Africa with high child mortality and very high adult mortality, and Sear-D, which comprises countries in South East Asia with high child and adult mortality.2^–6

Definition and selection of interventions

The term intervention is defined to include any preventive, promotive, curative, or rehabilitative action that improves health. Interventions are analysed individually and then in combinations or packages that could be undertaken together (box 1), taking into account interactions in costs or effectiveness, or both.

Interventions were chosen for analysis either because they are commonly used or because disease control experts have advocated their introduction. In each case, some evidence was needed that the intervention could be effective. The list is not exhaustive, and excluding an intervention does not imply it is cost ineffective.

All interventions and combinations are assessed assuming they are implemented for 10 years starting in 2000, the year the Millennium Declaration was signed. Good policy making would then require a reassessment of strategies. Costs, therefore, are only incurred over 10 years, but all benefits accruing because of actions taken between 2000 and 2010 are included.

Intervention costs

Costs are measured from the perspective of society as a whole, to understand how best to use resources regardless of who pays for them, or indeed, whether they are paid for at all. For example, village volunteers working for maternal health must be included because they could be working for some other health programme if they did not work for maternal health. All resources used for each intervention or combination are included and valued.

Data on the costs incurred by people to access services (such as travel costs) are rarely available, and we, like most other studies, have excluded them.7 8 We also excluded domestic taxes; from the perspective of the society, they simply transfer financial resources from one person to another and do not use up a physical resource such as capital or labour. The effect of interventions on the time and potential earnings of patients and unpaid carers—that is, work time lost—is a vexing question in cost effectiveness analysis but, as here, it is often excluded on ethical grounds. Inclusion would give priority to extending the life of people who earn more (box A on bmj.com).9 10

Costs are evaluated assuming 80% use of capacity (technical efficiency) for all interventions; this is relatively high but achievable. Use of a constant figure ensures that differences in cost effectiveness ratios are due to fundamental differences in the technologies or strategies adopted and not simply to an intervention being implemented poorly in a dysfunctional health system. Accordingly, the results provide practical information about the appropriate mix of interventions for various levels of resource, assuming interventions are implemented relatively efficiently.

Measles immunisation is one of the key interventions for child health

Credit: P VIROT/WHO

Classification and measurement of costs

Costs are divided into patient and programme levels. Patient level costs include face to face delivery by a health provider—for example, medicines, outpatient visits, inpatient stays, and individual health education messages. Programme level costs include all resources required to establish and maintain an intervention: administration, publicity, training, and delivery of supplies.

We used a standardised ingredients approach to measure costs, requiring information on the quantities of physical inputs needed and their unit cost. Total costs are quantities of inputs multiplied by their unit costs. To obtain programme level costs, we commissioned costing experts to collect data in 17 countries on the physical inputs (human resources, office space, vehicles, electricity, other services, and consumables) required to introduce and run an intervention using a standard template.11^–13 This was supplemented by information from programme managers in other countries known by WHO staff.

For patient level costs, quantities were taken from various sources. When published studies of effectiveness were available we used these to identify the resources necessary to ensure the observed level of effectiveness. In other cases, we estimated the resources required from the activities outlined in WHO guidelines for treatment.14^–17 Programme experts checked the face validity of all estimated quantities. We could not always determine the quantities of primary inputs (human resources, consumables), so we estimated quantities and prices at an intermediate level for several inputs: inpatient days at different hospital levels, outpatient visits, and health centre visits.

Unit costs for each input were derived from an extensive search of published and unpublished literature and databases along with consultation with costing experts. The full costing database is available at www.who.int/choice. For goods that are traded internationally, we used the most competitive price available internationally. For example, drug prices were taken from the latest WHO negotiated prices. A mark-up was included for transportation costs. For goods available only locally (human resources, inpatient bed days, etc) unit costs vary substantially within countries. We therefore ran cross country regressions using the collected data to estimate the average cost (with adjustments for use of capacity) for each setting.11^–13 18

Variations in scope and scale

Some interventions can share resources like vehicles, buildings, and administrative staff, so the costs of doing both together are less than the sum of the costs of the two individual programmes. We carefully identified possible shared costs when combinations of interventions were evaluated (box B on bmj.com).

Costs were estimated at three standard levels of coverage: 50%, 80%, and 95%. We assumed that interventions are first provided to, or obtained by, easy to reach populations. Coverage then expands to more outlying, sparsely populated areas. Facilities still need to be built despite the lower population density, so use of capacity will be lower, and the costs per patient treated correspondingly higher at higher levels of coverage. On the other hand, some interventions require a fixed number of staff, office space, or equipment to begin work. The same people and resources cope with increases in coverage so the programme level costs can be spread over a larger population, reducing the costs per person covered. Both types of effects are included in the costing exercise, and the net effect varies by intervention.

Costs are reported in international ($I) rather than US dollars with 2000 the base year (box 2). Future costs are discounted to 2000 values using a 3% discount rate.19 These costs do not translate directly into the financial or cash requirements to run or expand interventions. Estimates of the cash requirements are already available.20^–24 Here, we estimate the opportunity cost of all resources required to provide interventions, regardless of who pays for them, to explore the combination of interventions that makes the best use of these resources.

Assessing the effect of interventions

Interventions improve health through their effect on incidence, remission, severity, and mortality. We obtained efficacy data from (in order of preference) systematic reviews with meta-analysis; randomised studies, and before and after evaluations of country programmes. Effectiveness was obtained by adjusting efficacy by a factor between 0 and 1 to allow for less than perfect adherence to recommended practices. The adjustment factor was taken from the literature whenever possible, with expert opinion the last resort. Finally, the eligible population was multiplied by coverage to determine the total numbers benefiting from estimated effectiveness. No information was available on the effect of increasing coverage on health effects. We assumed, therefore, that the health benefit of each child treated for pneumonia (unit effectiveness), for example, did not vary with coverage.

It was rarely possible to obtain information on the joint effectiveness of multiple interventions implemented simultaneously. We therefore assumed that the health effect was additive for interventions that affect different health outcomes. For interventions acting on the same outcome the joint effect was estimated by multiplying the individual relative risks (box C on bmj.com).

Population health effects

We used a population model, PopMod, to project the effect of interventions on the aggregate healthy years of life lived by a population, combining incidence, prevalence, and mortality and estimates of disease severity with information on intervention coverage and effectiveness.25 The exception was for HIV and AIDS, for which we used a model that allows death rates to increase with time spent ill. Regional epidemiology was taken from the latest internal WHO update of the global burden of disease study.26

PopMod projects regional population over time, allowing people to move in and out of disease states, or die, in accordance with incidence and remission rates. To derive a single indicator of population health, time spent in each state is given a weight (health state valuation or disability weight) reflecting disease severity. Weights were taken from Murray and Lopez.27 The population is projected for the length of time necessary for people affected by the intervention to live their full life course. The difference between the aggregate number of healthy years lived by the projected population in the intervention and in the “do nothing” scenario is the population health gain attributable to the intervention.

The outcome indicator is essentially the number of healthy years of life gained by an intervention, also called the number of disability adjusted life years (DALYs) averted. The mechanics of estimating DALYs are virtually identical to those required to estimate quality adjusted life years (QALYs), although there are some differences in the interpretation (box D on bmj.com). The base case analysis discounts DALYs averted in the future at 3% and gives greater weight to DALYs averted during adulthood. This is relaxed in the sensitivity analysis.

Calculating cost effectiveness

Cost effectiveness analysis generally considers the costs and effects of adding new interventions to current practice or of replacing an existing intervention with another targeting the same condition. Here we evaluate the full set of existing interventions by first considering what would happen to population health if they all ceased to be implemented today. This is the null or do nothing scenario (box E on bmj.com). The current population is followed over time in PopMod assuming that all interventions cease, using the information on epidemiology, effectiveness, and the current coverage of interventions described earlier.

Next, we trace the implications for population health of adding all possible interventions singly and in various combinations, against the baseline of doing nothing. The difference is the gain in health due to the reduction in disease burden from the intervention(s) (DALYs averted). The costs of each scenario are then compared with the gain to identify the most cost effective set of interventions at different levels of resource availability. The comparison of the current mix against the optimal set for the resources currently available shows areas of inefficiency. The optimal set for higher resource levels shows what should be done if existing efforts to raise more resources are successful.

Interpreting results

For each paper in this series, the cost of every intervention and package is plotted against the corresponding population health effect relative to the do nothing scenario for that condition. We then determine the order in which interventions would be chosen and combined at given levels of resource availability if cost effectiveness were the only consideration. The graphical depiction we call the expansion path. To understand this process, two cost effectiveness ratios are defined. The incremental cost effectiveness ratio denotes the additional costs and benefits of a new intervention or package relative to what existed before, the previous point on the expansion path. Comparison of the incremental cost effectiveness ratio to doing nothing is sometimes called the average cost effectiveness ratio. Box F on bmj.com provides an example of how the expansion path is determined and interpreted.

Uncertainty

Cost effectiveness is an inexact science, particularly when data are limited and of poor quality. Many sources of uncertainty cannot be captured by statistical confidence intervals because no sampling is involved. Probabilistic uncertainty analysis has been developed for these circumstances, but technical limitations do not yet allow it to include the complex correlations inherent in combining the large set of interventions in this series (box G on bmj.com).28 29

Accordingly, the individual papers incorporate relatively simple forms of sensitivity analysis that are easier to understand. In the final paper, the results of all the disease specific analyses are integrated to develop priorities from the perspective of a policy maker responsible for all health millennium development goals, not just one. For this cross disease analysis, we believe it is not possible to recommend that an intervention shown to cost $45 per DALY averted is more efficient than one costing $60, given the nature of the uncertainties. However, we are much more confident that $45 per DALY is better than $450 per DALY. For the broad comparison across goals we, therefore, interpret cost effectiveness figures in broad order of magnitude ranges (box H on bmj.com).

Conclusions

Progress in the five years since the Millennium Declaration was signed has been disappointing.1 More funds to improve health in poor countries are urgently required, but money alone will not be sufficient to achieve the goals. Open discussion of ways of getting more from the available resources is required, both to allow countries to improve the health of their populations more quickly and to give potential donors confidence that new funds will be used effectively. Application of the methods outlined in this paper, as described in the next six articles in this series, will facilitate open debate and encourage changes in strategies where necessary.

This article is part of a series examining the cost effectiveness of strategies to achieve the millennium development goals for health

Editorial p 1093, Papers p 1107 and Education and debate p 1133