Appraising the value of evidence generation activities: an HIV modelling study

Introduction The generation of robust evidence has been emphasised as a priority for global health. Evidence generation spans a wide range of activities including clinical trials, surveillance programmes and health system performance measurement. As resources for healthcare and research are limited, the desirability of research expenditure should be assessed on the same basis as other healthcare resources, that is, the health gains from research must be expected to exceed the health opportunity costs imposed as funds are diverted to research rather than service provision. Methods We developed a transmission and costing model to examine the impact of generating additional evidence to reduce uncertainties on the evolution of a generalised HIV epidemic in Zambia. Results We demonstrate three important points. First, we can quantify the value of additional evidence in terms of the health gain it is expected to generate. Second, we can quantify the health opportunity cost imposed by research expenditure. Third, the value of evidence generation depends on the budgetary policies in place for managing HIV resources under uncertainty. Generating evidence to reduce uncertainty is particularly valuable when decision makers are required to strictly adhere to expenditure plans and when transfers of funds across geographies/programmes are restricted. Conclusion Better evidence can lead to health improvements in the same way as direct delivery of healthcare. Quantitative appraisals of evidence generation activities are important and should reflect the impact of improved evidence on population health, evidence generation costs and budgetary policies in place.


Detailed description of transmission model
A simple dynamic transmission model is used to model HIV transmission independently in nine regions within Zambia. The model is run over the period 1980-2030. The HIV interventions considered are late antiretroviral therapy (ART), voluntary male circumcision and early ART. Late ART describes the provision of ART for those who present for care, usually due to ill health, and who typically have CD4 counts below 350. Early ART describes the provision of ART to individuals who are identified via active outreach testing and who typically have CD4 counts above 350. Early and late ART reduce the probability of onward transmission of infection as well as resulting in direct health benefits to the individuals receiving treatment: early ART removes the risk of progression to late stage infection and late ART improves quality of life and reduces the risk of HIV-related death. Circumcision reduces the risk of men acquiring HIV infection.
The model structure is shown in Figure S1. States S1-S4 denote individuals susceptible to infection: S1 denotes circumcised men not at risk of infection, S2 circumcised men at risk of infection, S3 non-circumcised individuals at risk of infection and S4 non-circumcised individuals who are not at risk of infection. I1 denotes individuals in the earlier infection state who have CD4 counts above 350 and I2 individuals in the second later infection state who have CD4 counts below 350. EART and LART denote individuals receiving early and late ART respectively. Sex and variation in propensity for risky sexual behaviour are not explicitly incorporated in the model.
Individuals enter the modelled population at age 15. The number of individuals entering the population is denoted b. The proportion of individuals entering the population who are not at risk of infection (τ), and the proportion who are circumcised (c) determine how individuals are distributed across states S1 to S4 at model entry. The model is calibrated to a set of region-specific estimates of HIV prevalence for the year 2013. This is achieved by identifying the value of τ that minimises the sum of squared differences between the estimates of HIV prevalence and the model prediction of HIV prevalence for each region. This allows differences in sexual behaviours across regions to be reflected in the model. λ C and λ NC describe the force of infection in circumcised and uncircumcised individuals, respectively. σ describes the rate of progression from infection state I1 to I2. Individuals face different mortality rates depending on their health state. All individuals face a risk of leaving the sexually active population of μ G , individuals with early stage infection face an additional risk of death (at rate μ E ); as do individuals receiving late ART (μ L ) and those who are untreated and in the more severe infection state (μ I ). The birth rate is (μ G +g) where g represents the population growth rate. and denote the proportion of those eligible for early and late ART who receive treatment, respectively. In addition the dashed arrows indicate one-off transitions possible at discrete points in the model. These are used to reflect immediate scale up of interventions as discussed below. The force of infection is described by the following equations: ( 1 + 2 + 3 + 4 + 1 + 2 + + ) where is the reduction in the risk of acquisition of HIV in individuals who have been circumcised, is the rate of transmission and is the reduction in the rate of transmission from individuals receiving early or late ART. During the intervention period (2015-2030) a proportion ( ) of those progressing from early to late stage infection receive ART. This proportion can range from 0 to 80%, as up to 80% of individuals with late stage infection can be identified and are willing to undergo treatment.(1) A proportion ( E ) of newly infected individuals and individuals infected prior to 2015 receive early ART (the latter shown by the dashed arrow from I1 to EART). This proportion can range from 0 to 73% as up to 73% of individuals with early stage infection can be identified and are willing to undergo treatment.(2) A target circumcision rate is set and at the beginning of the intervention period individuals are moved from state S4 to S1 and from S3 to S2 to meet this target, as shown by the dashed arrows. The number of people who make the transition from S3 to S2 ( ) and S4 to S1 ( ) are: where ϕ C, is the desired increase in the proportion of individuals circumcised. We assume that it is possible for up to 40% of all individuals to be circumcised (i.e. approximately 80% of all males corresponding to half of the modelled population). The values taken by , therefore depend on the region-specific baseline level of circumcision, c. For example, if the baseline level of circumcision is 25% can take values of between 0% and 15%.
Any combination of these coverage levels for circumcision, early ART and late ART can be selected by choosing the corresponding values for , and . The way in which coverage levels are selected is determined by the resource allocation component of the model, which is discussed in the next section.
Each health state is associated with a different health related quality of life weight to reflect differences in the morbidity of individuals in the model. Costs are incurred for each circumcision, each year of ART and each HIV test required. Early ART involves active outreach to identify HIV positive individuals in the community, and therefore incurs a testing cost. The number of tests is calculated by multiplying the numbers of individuals initiating early ART by the number of individuals who must be tested to identify each HIV positive individual.
Costs depend upon the scale of intervention provision due to both the need to employ additional outreach measures, such as travel vouchers, to achieve higher coverage levels, and economies of scale. We follow the approach taken by Meyer-Rath et al. in order to quantify these effects.(3) If total coverage exceeds 40% costs increase by a percentage (v) to reflect outreach activities. This percentage increase depends upon the coverage level and elasticity of demand (e): Outreach costs are calculated independently for early and late ART as these interventions relate to distinct populations of individuals. Increasing returns to scale are modelled for a proportion (α) of total costs. These costs respond to scale of provision according to a scale elasticity of γ, where γ=0.5 implies that a 10% increase in the number of individuals receiving an intervention results in a 5% increase in costs. The returns to scale associated with ART depend on the total number of individuals receiving late or early ART.
The parameter values informing the transmission model are shown in Table S1.
Uncertainty in the following key parameters is reflected in the model: prevalence in the year 2013 in each region; the effect of ART and circumcision on transmission ( , ) and the cost of circumcision and ART therapy. All other parameters within the model are held fixed. Parameter uncertainty is propagated through the model via Monte Carlo simulation, this involved randomly sampling from a distribution assigned to each uncertain parameter and running the model for each set of sampled parameters, 600 simulations were used. Each simulation represents one 'realisation' of uncertainty or 'state of the world' that could occur. Each parameter is sampled independently across regions to reflect differences in epidemiology and ability to deliver care effectively and efficiently.
The model provides estimates of total costs and total health outcomes, expressed as quality adjusted life years (QALYs), for each region.  Figure S2(a). The standard error or confidence intervals for the DHS estimates were not reported by region, making it difficult to assess whether the estimates overlap or not. Any differences noted between the two sources will be due to the structure of the model and assumptions. Importantly, the model is not intended to accurately reflect HIV prevalence and epidemiology in Zambia, or to inform decision making in Zambia. Instead the model has a set of geographic units with a range of epidemiological conditions and magnitudes of epidemic that is consistent with several real countries, including Zambia The uncertainty around the prevalence estimates included in the model in each region is based on a beta distribution as shown in Figure S2(b).

Detailed description of resource allocation model
We represent resource allocation as a two-stage decision making process. (17) The first stage involves planning coverage levels for each intervention in each region, and the corresponding resources that will be allocated. Decision makers are assumed to base their plans on the expected costs and health benefits of all possible ways in which the resources could be allocated. We assume that decision makers will select the resource allocation that maximises health across regions subject to the national HIV budget.
The expected costs and health benefits used in this process reflect the average costs and health benefits across all possible eventualities that could occur (or across all 'realisations' of uncertainty). In reality, only one of these eventualities occurs and the decision maker must react to this. If they don't and instead stick to their original plans then they may end up exceeding allocated budgets or being left with unspent funds. We therefore model a second stage in which decision makers respond to the realised eventuality. These two stages are described in detail in the following sections.
We run resource allocation model for a range of budgets. Resch et al (18) have estimated total AIDS expenditure for 12 low-and middle-income countries (Botswana, Côte d'Ivoire, Ethiopia, Kenya, Mozambique, Namibia, Nigeria, Rwanda, South Africa, Tanzania, Uganda, and Zambia). These countries account for 52% of AIDS cases worldwide. Using the most recent data available for each country (which ranged from 2009/10 to 2012/13) in combination with population data from the World Bank (4) implies a total annual HIV budget of $3-$175 per capita. Ten of the twelve countries had a total annual HIV budget of $3-$32 per capita. The remaining countries -Botswana and Namibia -had much higher annual budgets of $175 and $136 per capita respectively. We therefore explored annual budgets of $3-32 per capita, scaling these up to reflect the total population in our example at the start of the intervention period and the 15 year time horizon. This resulted in a budget range of $0-$3,000 million. We present results in this paper for budgets of $0-$2,200 million as at budgets of $2,200 million all modelled investment opportunities had been exhausted.

Planning the HIV investment strategy
We constructed an optimisation model to identify the allocation of resources across geographical areas and interventions that maximises population health. This model selects coverage levels for each intervention ( , and ) in order to maximise total health benefits (QALYs) subject to expected total costs being within the HIV budget. As the impact of the interventions is interdependent we model each possible combination of coverage The objective function specifies that the goal is to maximise total health across regions. The optimisation process selects values for the decision variables ( ) that will maximise total health subject to three constraints: (1) each decision variable must take the value 0 or 1 i.e. an intervention set can be selected or not; (2) within each region only one intervention set can be selected; and (3) the total cost of the selected intervention sets across regions must not exceed the national budget (B).
In addition, the decision maker faces a constraint that universal access to late ART must be offered to all those who can feasibly be reached for treatment (80% of all late stage infections (1) The first component of the additional constraint is: where denotes an indicator function that takes a value of 1 if the condition is true and zero otherwise. This constraint requires that the all regions must have implemented late ART at 80% (first term) if a region has implemented early ART or circumcision (second term).
The second component requires that coverage of late ART is equal across regions: This is achieved by requiring the average late ART coverage (first term) to equal the regional late ART coverage (second term) for every region.

2.2
Responding to the realised state of the world We model a series of different policy responses to each realisation of uncertainty. As we do not know which realisation will occur, the outcomes associated with each policy are calculated for each realisation and then averaged across all realisations.

Regional policy under current information
Under the regional policy under current information each region is required to remain within their planned regional budget. To achieve this spending on each intervention in each region is maintained at the planned level. This does not guarantee that regions will achieve the coverage they planned. This is illustrated in Figure S4 which shows the response to uncertainty for a single intervention in a single region. Based on expected costs and effects the decision maker plans to implement 60% coverage at a cost of $1 million. Panel (a) of Figure S4 shows that if actual costs are higher than expected (e.g. due to a larger number of individuals presenting for treatment, or higher than expected treatment costs) the decision maker must cut coverage to 30% to remain within budget. When actual costs are lower than expected, coverage is expanded ( Figure S4 panel (b)). If the available budget exceeds the amount required to fund the maximum possible coverage (here 80%), then the difference between the available funds and the cost of achieving maximum coverage is wasted.

(a) Response to actual costs exceeding expected costs (b) Response to actual costs falling below expected costs
Figure S4: Regional budget policy (one region, one intervention): Circles indicate planned coverage, squares indicate realised coverage. Panel (a) shows that when costs exceed those expected coverage must be reduced. Panel (b) shows that when costs are lower than expected coverage can be expanded. If the original planned spending exceeds the cost of maximum coverage (here 80%) funds will be wasted.

Perfect information
It is possible to improve the information available to inform resource allocation decisions. For example, large surveys could be conducted to improve our understanding of regional epidemiological conditions. We assess the maximum possible value of improving the information available to decision makers by looking at the health we could generate if we had 'perfect' information. If we had perfect information we would know exactly which eventuality would occur and could perform the initial optimisation described in Section 2.1 using observed costs and benefits (rather than expected costs and benefits). To establish the value of perfect information we therefore run the optimisation analysis under each realisation of uncertainty and average the results across all realisations of uncertainty.

National policy under current information
Under the national policy plans are only revised to the extent that national HIV costs under-or over-run the national HIV budget. Under this policy funds are fungible across interventions and regions to support planned coverage levels. Any surplus at the national level is used to increase funding beyond that required to meet the original plans by the same proportion for each intervention in each region. Any deficit at the national level results in the funds required to achieve planned coverage being cut by the same proportion.
The implications of the national policy are shown in Figure S5 where we have two interventions and one is highly cost-effective (A) whereas the other is less cost-effective (B). Under the regional policy the decision maker plans to implement 60% coverage for both interventions but actually delivers coverage of 45% for A and 70% for B ( Figure S5 Panel (a)). Under a national policy ( Figure S5 Panel (b)) plans are identical to those made under the regional policy. Although the cost of interventions A and B are not as expected, the total costs are within the originally planned total cost. Funds can therefore be re-allocated between regions to achieve the original planned coverage levels of 60% for intervention A and B.

Contingency fund policy under current information
When a contingency fund is used, plans are made based on the total HIV budget less the amount dedicated to the contingency fund. Plans are therefore more modest than under other policies. The contingency fund exists to avoid programmes being cut back due to funding issues. If there are sufficient funds to pay for the planned programme then this policy works in exactly the same way as the regional policy though with a lower budget. When there are insufficient funds to pay for the planned programme the contingency fund can be used. If the contingency fund is insufficient to pay for all programme over-runs, a reduced claim is met (and claims are reduced by the same proportion across all programmes).
If implemented for one intervention in one region the contingency fund will always result in lower coverage and health than the regional policy. This is because when a contingency fund operates spending on the intervention will always be equal to, or lower than, spending under the regional policy. 1 However, when there are multiple interventions or regions the contingency fund can result in improved health compared to the regional policy. Under the contingency fund 20% of the total $3m budget ($0.6m) is set aside as contingency ( Figure S5 Panel (c)). This results in a lower investment in intervention B (as this is less cost-effective than intervention A) which has a planned coverage of 40%, and an actual coverage of 45%. The cost over-run for intervention A is met by the contingency fund, allowing planned coverage to be preserved at 60%. In this instance the contingency fund may improve population health compared to the regional policy as it supports coverage of the more costeffective intervention. This will not be the case in all realisations of uncertainty. The contingency fund may produce less health than the fixed regional budget policy under a range of conditions. If there are under-runs in a number of programmes this will mean that the contingency fund is partially or completely unspent. If the contingency fund preserves planned programmes in states of the world in which they are particularly costineffective (e.g. due to high costs) then this would also reduce health. Finally, if the contingency fund results in plans that involve maximum coverage for more interventions this will increase the likelihood of wasted funds.
(a) Regional policy

Implementation of simulation modelling
Estimating the health generated under perfect information is relatively straightforward. To estimate the health expected under perfect information the optimisation process is run for each realisation of uncertainty and the corresponding health is recorded. The expected health is then estimated as simply the average across all realisations of uncertainty. The implementation is more complex for the current information scenarios. In each of these scenarios the response to uncertainty is either to preserve or adjust spending. There is therefore a pattern of spending on each intervention in each region that we expect for each realisation of uncertainty. However, spending on each intervention is not an input in the transmission model. A matching process is therefore used to identify the intervention set that most closely produces the spending pattern we expect (in each region). This allows the health associated with the spending pattern to be estimated. Again, this is estimated for each realisation of uncertainty, and then averaged across all realisations of uncertainty.
The transmission model is run for all intervention sets defined by evaluating interventions at 1% increments (ART interventions) and 2% increments (circumcision intervention) in coverage levels ( = 0%, 1%, … ,80%; = 0%, 2%, … ,40%; and = 0%, 1%, … ,73%). In total 124,173 intervention sets (81*21*73) are therefore evaluated in each region and for each realisation of uncertainty. The exact spending pattern of interest will generally not have been observed in this set of transmission model runs. The intervention set that most closely matches the required spending pattern is therefore identified using a two stage process. Firstly, the set of 'affordable' interventions are identified. Affordable interventions are those which offer spending on each intervention component (late ART, circumcision and early ART) that is less than or equal to that of interest. Secondly, amongst these affordable intervention sets the absolute difference between spending on each intervention and the spending expected is calculated. These absolute differences are then summed across the three interventions and the intervention set with the smallest sum of absolute differences is selected as the best match. In some regions of Zambia the proportion of individuals entering the modelled population who are already circumcised is relatively high (see Table S1). Under all policies the maximum intervention level for circumcision is set so that the total population coverage for circumcision cannot exceed 40% for all individuals (i.e. 80% for men), this results in variation across regions in the extent to which circumcision coverage can be increased by the HIV programme.