Perceived validity of findings

Perceived validity of findings was influenced by accordance of findings with expectations, organisational beliefs and alternative evidence. Model outcomes that confirmed existing assumptions or other evidence were used to justify spending resources and strengthen funding applications:

There were no surprising findings, but it validates the intuitive thinking. (IFHO)

Where findings corroborated global guidance or institutional beliefs, Optima added to a local evidence base and increased motivation for using the findings in resource allocation.

The finding of the Optima exercise has strengthened that thinking and that expectation. (CS4)

For instance, in one country Optima findings were actively used in discussion with government bodies to scale-up opioid substitution therapy (OST), helping to overcome legislative barriers to OST prescription and gain multidepartmental support for OST. Despite there being considerable existing evidence for interventions such as OST, stakeholders valued the ability of Optima to contextualise global evidence to the local setting.

Conversely, in some settings where results challenged expectations or beliefs, the modelling evidence was discounted or not considered for financial decision-making, such as an example where voluntary medical male circumcision (VMMC) was not prioritised. In certain cases respondents communicated dissatisfaction with the choice or handling of data used in the model, despite commonly reporting involvement in data collation. In the above example, stakeholders had queried the VMMC programme cost data used in the model. In one setting with pronounced uncertainty on population size estimates, stakeholders believed that the proportion of new HIV infections among people who inject drugs (PWID) had been overestimated in the Optima model and thus resisted implementing recommendations to scale-up PWID prevention interventions.

The last two rounds surfaced a negative factor for [country]: 50 percent of the new HIV cases come from IDUs [people who inject drugs], which in reality is not the case. We did send our remarks emphasizing that we do not have such a high rate among IDUs. (CS8)

Perceived validity was also compromised in select examples when there were discrepancies between Optima HIV and epidemiological projections from other HIV epidemiological models used for annual planning and forecasting.

Diverse stakeholders emphasised that the policy environment is increasingly structured around cross-cutting programmes, including structural interventions and integrated disease programmes. Respondents portrayed that reflecting these elements in modelling by aligning analyses to country targets and priorities would facilitate broader utilisation in policy decisions.

In some settings respondents conveyed that there was a general scepticism of modelling among some personnel involved or key decision-makers due to perceived subjectivity and potential for biases during model calibration. This was not unique to Optima but had led to some stakeholders being critical of modelling evidence. One country representative explained that evidence generated through modelling was perceived as inferior to empirical data derived from programmes or intervention studies.

When they see the back of it and appreciate that they can play with the figures, then it becomes more [of a] political instrument rather than [an] epidemiological instrument. (IFHO)

Participants described resistance to findings among some stakeholders when a specific population group would be disadvantaged by implementing Optima recommendations due to consequences for health and social equity, including health impacts outside of HIV/TB, if funding were reduced.

We faced some challenges and resistance while reallocating those funds, specifically on the side of civil societies and NGOs because due to the Optima results, we had to taper off funding from one community and groups towards the others. (CS2)

Respondents discussed a few means to strengthen perceived validity of Optima and modelling processes. Most eminently, extended training and building local capacity to use Optima were portrayed as a key route to developing local ownership and building local skills and capacity to support, interpret and independently use modelling studies. The training provided from Optima was appreciated in many settings:

In terms of training, I think it was useful. For each country group there was a representative delegated by your institution… It was quite interesting and efficient. (CS8)

However, respondents from one country conveyed that no training had been received, and respondents from multiple countries suggested that more training would have been useful, particularly for technical leads.

Need to increase the ownership of Optima here locally through a greater amount of trainings. Yes, we have gone through the five day long workshop, but it was quite intensive yet still not enough. We need more hands-on training later on. Optima math modeling is quite sophisticated, and not something you will get fully in five days. And what we need essentially here is the experts or resources who would be capable of using Optima on their own, to the extent possible. So more training would be required. (CS2)

Funding organisation representatives favoured in-person workshops for training and modelling, suggesting they offered a more immersive experience and facilitated meaningful involvement and dialogue.

Beyond training, familiarity with Optima and allocative efficiency was achieved through involvement in repeat Optima analyses or other HIV investment case projects and facilitated more meaningful engagement with the processes, including the critical assessment of data inputs and findings and enabled greater ownership.

Stakeholder engagement

The extent, timing and type of stakeholder engagement was a major theme influencing the uptake of modelling findings. International representatives perceived that effective engagement with in-country stakeholders, and particularly the MOH, was integral to fostering local ownership of Optima products and required concerted effort and time to achieve.

This component of ownership is probably the hardest, and it’s the one that was emphasized a lot. It started with a letter to the Ministry of Health from us. And then it was, when you own it, you designate that this is your report. And then they validate the reports, and so on. And some cases it took quite a long time to validate the final reports, but we got to it… Despite all of the other priorities that were running. So in that sense, I don’t think that any country will tell you ‘they don’t know’, ‘we didn’t do it, it’s something that [IFHO] did’. It’s not the [IFHO]–they did it. (IFHO)

Although representatives from the NAP, NTP and/or MOH usually led country involvement in Optima, country respondents generally appreciated engagement of a wide range of stakeholders, spanning technical and policymaking roles and multisectoral representation from non-governmental organisations and civil society organisations (CSO), funding partners and UNAIDS. Representatives from one country emphasised the importance of discussing findings with ministries outside of health to gain broader support for interventions such as OST and to influence national strategy.

The joint oversight committee meeting is the one that is critical because the stakeholders, or the members rather, are the CEOs and country directors of all of the stakeholders in the HIV/AIDS response. And therefore those are the decision makers of allocation of funds. (CS1)

It would be beneficial to include someone from civil society in this taskforce. If excluded, they might think the government is hiding something or has some sort of hidden agenda. We will gain more support and understanding from them if that person will be included. (CS2)

In numerous settings, effective joint planning between country partners and the Optima team supported alignment with country needs and the actioning of study recommendations.

It was the NTP that proposed the modeling exercise should be done at district level. (CS4)

It’s not that we say ‘you should invest more in treatment because we are the [IFHO]; We say, ‘we together ran the analysis, here is the science behind it. This is how much money we need to hit the 90-90-90…It helps to have, you know, ‘we’ve done this together, this is what the data says, so this is probably where we should move’. (IFHO)

However, there were differing views on whether policy and programme leads, including government officials, should be involved earlier and throughout the analysis to promote ownership of findings, or whether it was effective to limit engagement to stages such as defining national policy questions and dissemination. In some settings international funding representatives emphasised that they intentionally focused participation on monitoring and evaluation (M&E) personnel to promote Optima as a data-driven technical exercise and avoid findings being perceived as political.

We sold it as a very technical exercise. And we involved people from the M&E departments. There were no decision makers, just the people who do the data, the M&E. So the result was it was a very helpful exercise… (IFHO)

In contrast, stakeholders from one country felt that limiting involvement of relevant international funding entities to dissemination may have inhibited the appreciation and uptake of findings, suggesting earlier engagement may have been more effective.

They were hearing [of Optima analysis] for the first time and being on the tail end of it. Which might have been a little too much to chew at one sitting…It would work better if we get them through the mill earlier on before we get to the results. (CS1)

Participants valued meaningful opportunities for dialogue, and in isolated cases respondents recalled insufficient opportunity to provide feedback, thus limiting the perceived validity and subsequent application of findings.

We were not able to follow through the whole process because we were not in a position to give feedback to our principles [country leads]. (CS3)

In one setting, respondents attributed the lack of ownership to minimal follow-up, discussion and influence of Optima results on policy.

It’s an important factor, that the [name of institution], which was the key institution, to own the results of this analysis-modeling exercise. The [Institution] did not really promote Optima. Has not brought up Optima recommendations in discussions about the Global Fund Proposal. Or the PEPFAR funding. Or the National AIDS Strategy. I still think they have not released this document as final. (IFHO)

Timing and opportunity

Timeliness of Optima analyses influenced research utilisation by impacting availability of data to parameterise models and opportunity to inform key strategy or funding decisions. In some cases, strategic planning was successfully informed by preliminary findings.

The exercise was still being done during the strategic plan development and also the NFM [New Funding Mechanism] grant making. I think this was the major time when there has been key decision making and resource allocation. Since we had some preliminary output from Optima, I think those recommendations were also being considered and incorporated in those kind of exercises. (CS4)

Both country and funding organisation representatives spoke about the importance of factoring in sufficient time for preparation, country engagement and data collation and to anticipate timing of most recent epidemiological or financial studies. Representatives from one country expressed poor timing of analytical activities due to inadequate integration of various modelling and costing exercises. This led to a duplication of effort to track resources and collate data, such as for National AIDS Spending Assessments, National Health Accounts, Optima and Spectrum.

Competing demands at the time of Optima analyses and busy agendas were constraints on stakeholder engagement. In one example, the Optima application happened simultaneously with the development of the M&E framework for the new National Strategic Plan, thus limiting staff availability to support the Optima analysis.

In the context of [country name] …there are lots of conflicting priorities. … The concept of efficiency was not high on the agenda. The other problems they are facing, in their view, are more important. No human capacity or time to focus. (IFHO)

In some cases, stakeholders recognised that competing priorities coupled with limited resources increased the perceived value and application of allocative efficiency analyses.

The issue of allocative efficiency always remains a priority in terms of our response because we don’t have adequate resources and we also do have now quite a number of competing priorities. (CS3)

Health system financing mechanisms

Respondents from several settings identified that funder preferences had a bigger influence on resource allocation than Optima findings. The two were not necessarily exclusionary, and funder support may have reinforced the scale-up of interventions such as HIV self-testing, which was prioritised in several Optima analyses. Elsewhere respondents conveyed that pre-exposure prophylaxis was being scaled-up, despite this not being supported by model findings, due to pre-established funding and government priorities.

Country representatives and funding organisation representatives both relayed that budgets tended to favour historical allocation patterns, which limited utilisation of research findings in policy and financing decisions in some settings.

Funding for subsequent years would usually be predominantly based on the previous year. So it’s more of a recurrent budget and budget proportions, not necessarily guided by evidence of impact. (CS1)

In some settings budget reallocations were constrained by the health system financing model. Systems that used line-item budgets (Line-item payment systems define the total amount to be financed to a hospital or other organisation based on the expected costs of clinical and non-clinical staff, equipment, medicines, utilities and maintenance, which are determined based on expenditure in the previous year)31 to finance inpatient services lacked the flexibility and incentive to transition to ambulatory, decentralised care models, even when Optima findings demonstrated alternatives to be more cost-effective and efficient. Without changing how health services are funded at a structural level, adopting these changes would lead to reductions in hospital funding in subsequent years. In these cases, the system lacked appropriate mechanisms to transfer funds, such that savings could not be reinvested in alternative activities nor for comparable care outside of the hospital setting.

The number of beds have decreased, but this does not mean that they release the funds … for other activities. (CS5)

Fragmented funding of epidemic responses also posed challenges to redistribute either existing funding or cost-savings through implementation efficiencies. Respondents from several settings spoke of difficulties in transferring cost-savings from general population HIV testing programmes to key population prevention and testing due to different sources of programme budgets.

Of course, for [country name] Government it is easier to fund HIV treatment and expenditures related to treatment, care and support for HIV positive people [sic]. The Government covers around 80 per cent of such expenditures…In terms of the prevention programs … the cost sharing in this case is the opposite: 80 per cent will come from Global Fund. (CS7)

In one case this meant that while the state budget increased commitments for key population prevention in line with Optima recommendations, eventually these funds were not disbursed due to the complexity in actualising resource reallocation. Subsequently, health system reform was identified as a potential enabler of resource allocation in line with modelling recommendations.

Respondents from one setting discussed that social contracting was being introduced to reduce the dependency on international funding, promote sustainability and improve the efficiency and quality of care. This enabled public health centres to purchase HIV prevention services through non-government and CSOs. The respondent felt that making the government the main recipient of international funding gave them more power to advocate to local governments and increase funding for key population programmes through social contracting, in line with modelling recommendations. They described this as a lengthy process taking place over 2–3 years which eventually enabled better adoption of allocative efficiency findings.

So, [local governments] begin to see the benefits of these expenditures now as well as the purpose of it. Thus it is much easier to convince them now that funding for these areas should be increased. So, the role of the main recipient as a main driving force in advocating state social contracting is very strong. (CS7)

Optima commonly engages most closely with the NAP and NTP for HIV and TB studies, respectively. Some respondents referred to these parties having limited influence on budget decision-making. Their actions were constrained to making recommendations to the relevant government structure, but this was insufficient to enable the application of Optima findings to resource allocation in some settings.

The NTP is not the owner of the government funds. It has limited influence on how the funds are allocated. (CS5)

Country teams often found it difficult to differentiate direct from indirect spending (eg, management costs and overheads). The latter is generally considered a fixed expense in Optima analyses, but some respondents reported a need to better understand indirect spending, such as administrative costs and supply chain management, to inform resource allocation.

The performance of a program depends on various factors, which I believe need to be included in future modeling to bring robust results…For instance, we have a shortage of human resources – it’s one of the determinants of the performance. (CS6)

Sociopolitical context

Respondents considered that political will was important when findings concerned key population groups or migrants, which were politically sensitive in some contexts. In settings where related behaviours are criminalised or these groups are not prioritised in existing resource allocation, Optima findings were more likely to face resistance from policymakers, particularly outside of the NAP or NTP or at a subnational level. This was particularly pertinent in settings in the process of transitioning from international to domestic spending.

A governor in a region may have never heard before how MSM [men who have sex with men] are influencing pandemics and now he is told that funds need to be allocated for condoms for this group, for lubricants. Usually, these things come as a shock to local government. (CS7)

Despite the expected resistance in these settings, funders perceived that Optima supported advocacy by providing objective rationale for prevention programmes for key populations. Optima provides an opportunity to ‘put the science before the politics’ (IFHO), and in this way has the potential to depoliticise resource allocation.

External factors such as sociopolitical unrest also impacted the influence of Optima findings on resource allocation. In one setting, conflict and the existence of autonomous regions meant that subnational HIV programmes and services are not controlled or funded by the government, and thus not influenced by government resource reallocations. The COVID-19 pandemic placed strain on healthcare resources and budgets, necessitating diversion of funds from other health programmes: ‘Every free cent was reallocated to COVID’ (CS8). In some settings, respondents reported increases in service costs due to COVID-19, limiting the relevance of prior allocative efficiency analyses.