Article Text
PDF
Abstract
In resource-constrained environments, priority setting is critical to making sustainable decisions for introducing new and underused vaccines and choosing among vaccine products. Donor organisations and national governments in low-income and middle-income countries (LMICs) recognise the need to support prioritisation of vaccine decisions driven by local health system capacity, epidemiology and financial sustainability.
Successful efforts have supported the establishment of National Immunisation Technical Advisory Groups (NITAGs) to undertake evidence-informed decision making (EIDM) in LMICs. Now, attention is increasingly focused on supporting their function to leverage local expertise and priorities. EIDM and priority-setting functions are complex and dynamic processes. Here, we report a pilot of a web-based decision-support tool. Applying tenets of multicriteria decision analysis, SMART Vaccines 2.0 supported transparent, reproducible and evidence-informed priority setting with an easy-to-use interface and shareable outputs.
The pilot was run by the Uganda NITAG who were requested by the Ministry of Health (MOH) in 2016 to produce recommendations on the prioritised introduction of five new vaccines. The tool was acceptable to the NITAG and supported their recommendations to the MOH. The tool highlighted sensitivity in the prioritisation process to the inherent biases of different stakeholders. This feature also enabled examination of the implications of data uncertainty. Feedback from users identified areas where the tool could more explicitly support evidence-to-recommendation frameworks, ultimately informing the next generation of the platform, PriorityVax.
Country ownership and priority setting in vaccine decisions are central to sustainability. PriorityVax promotes auditable and rigorous deliberations; enables and captures the decision matrix of users; and generates shareable documentation of the process.
- vaccines
- health policy
- immunisation
- public health
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Summary box
In resource-constrained settings, there is a pressing need to support evidence-informed priority setting regarding decisions addressing the introduction of under-used and new vaccines and to select between vaccine products.
The SMART Vaccines 2.0 platform was piloted in Uganda where five vaccines were under consideration for inclusion in routine immunisation.
The SMART Vaccines 2.0 tool was found to be supportive, and feedback from real-world use has streamlined the user experience to avoid onerous epidemiological assumptions and data burden.
The most updated version of the tool, PriorityVax, fulfils a need to support the deliberative process and is especially useful in highlighting the sensitivity of decisions to inherent bias and uncertainty in the evidence.
Introduction
Twenty years of cofinancing from Gavi, the Vaccine Alliance (Gavi) has greatly expanded access to underused vaccines and accelerated the introduction of vaccines in low-income and middle-income countries (LMICs).1 For national governments, introducing a new or underused vaccine can double or triple the cost of routine schedules and additionally commits a country to generations of spending.2 As new vaccines and formulations become available, the complexity of decision-making and resource allocation continue to grow. Furthermore, there can be tacit pressure to introduce vaccines,3 both international and domestic, to meet health and development goals. It is, therefore, unsurprising that national-level decision-makers, Gavi and other donor organisations seek to ensure that existing immunisation programmes are sustained while expanding disease prevention through new vaccine introduction.
To be eligible for Gavi funding, countries must account for a transparent, independent evidence-informed process to support their applications for financing. Ideally, this process should have three components to promote sustainable decisions: an independent body of representative stakeholders providing advice to the Ministry of Health (MOH); the gathering and evaluation of a relevant evidence base; and prioritisation among the options considered.
Priority setting is a specialist task4 that extends beyond the assessment of evidence because it requires comparison among multiple options across competing interests (eg, healthcare specialists, health policy leaders and programme and financing decision-makers). Guiding frameworks for priority-setting recommend the following elements, which are complementary to an evidence-informed decision-making (EIDM) process: (1) stakeholder engagement, (2) use of an explicit process, (3) information management, (4) consideration of values and context, and (5) revision or appeal mechanism.3 5 Systematic support for priority setting within National Immunisation Technical Advisory Groups (NITAGs) or MOHs in LMICs has been limited.6–9
Substantial efforts to assist LMICs with the identification of within-country expertise and the establishment of NITAGs10 followed from the first strategic objective of the WHO’s Global Vaccine Action Plan 2011–2020.11 These independent, expert bodies are expected to review of scientific evidence and data representative of the local context to inform and develop recommendations to support country ownership of policy decisions.12
Supported by Gavi and the Bill and Melinda Gates Foundation, a number of international organisations have offered capacity-building support to establish NITAGs. While greater emphasis was initially placed on fulfilling process indicators for establishing NITAGs, more recent efforts have sought to advance functional capabilities associated with EIDM, most notably by Agence de Médecine Préventive (AMP), the International Vaccine Institute and The Sabin Vaccine Institute.13 14 These programmes have additionally leveraged technical assistance from WHO and its regional offices, PATH and the US Centers for Disease Control and Prevention.15 16
These efforts have been important to advancing and expanding vaccine coverage in LMICs. Nonetheless, both Gavi-eligible countries and those graduating from cofunding have reported variable success in sustaining immunisation programmes.17 18 Efforts that bring clarity to how interventions align with available resources and population health priorities are most likely to promote the local ownership of decision making that is key to sustainable programmes and policies.19–21 Indeed, a review of European NITAGs reported that more than half the NITAGs (16/27) felt structural differences between countries (eg, the role of the NITAG, healthcare systems or epidemiological situation) meant recommendations would not be applicable elsewhere.22 Consequently, although there was support for sharing evaluation of evidence, even in middle-income and high-income settings there was recognition that recommendations for programmes and policies are highly context specific.
In settings where health needs are often unmet and health systems are under greater resource constraints, enabling locally contextualised, prioritised decision making is all the more important. Prioritisation is likely to be increasingly relevant to countries that graduate from donor support (eg, Gavi) and therefore take on the full economic cost of immunisation.18 23 Here, we describe the real-world use of a priority-setting tool both describing the piloting of the tool and discussing key points of the deliberations that arose. The latter will be further described elsewhere.
The call for priority setting in Uganda
In 2016, the Uganda MOH approached its NITAG, which was established 2 years earlier, for recommendations on which of five available new or reformulated vaccines should be introduced into the immunisation schedule: meningococcal A, hepatitis B vaccine birth-dose, switching from tetanus toxoid to tetanus-diphtheria, introducing routine yellow fever vaccine, and a second dose of a measles-containing vaccine. Further, the MOH was interested in prioritisation among those vaccines recommended for introduction.24 The need for prioritisation, rather than simply a go/no-go recommendation, was made clear by the challenges facing Uganda’s vaccination portfolio. A projected 90% financing gap (over 2016–2020) of vaccination programmes when subsidies were excluded25 add to challenges of access and delivery of vaccines,26 health facility readiness27 and are further complicated by Uganda’s refugee population28 and porous land borders with countries with endemic and zoonotic diseases, for example, meningitis, yellow fever and Ebola virus.29–32
Here, we describe the experience of the Uganda NITAG (UNITAG) in piloting SMART Vaccines 2.0—a tool to support vaccine-related priority-setting decisions—that was conducted in parallel with their existing evidence-to-recommendation (ETR) framework.
Priority-setting frameworks
The UNITAG sought technical assistance from AMP’s Supporting Independent Vaccine Advisory Committees (SIVAC) Initiative,14 and engaged in piloting the SMART Vaccines 2.0 platform supported by the Fogarty International Center at the US National Institutes of Health (NIH). A description of the NITAG process is given elsewhere.24 33
SIVAC framework
A number of decision-support tools exist that were either built for NITAG use or can be adapted to that purpose. However, underlying any tool and the related decision-making process are generic skills required to gather and interpret evidence. In addition to helping establish and strengthen NITAG secretariats, SIVAC produced documentation and ran workshops to assist with formulating evidence-informed recommendations.34 Its framework, used by the UNITAG, was based on WHO practice, using the Population, Intervention, Comparator, Outcome (PICO) method to formulate precise questions to inform systematic literature reviews. This identified evidence unique to each vaccine-disease scenario under review. The collated evidence was then subjected to an appraisal of quality, the UNITAG used a GRADE framework (Grading of Recommendations Assessment, Development and Evaluation).35
The SIVAC framework was designed to identify, gather and evaluate evidence. Its application to prioritisation, including a process by which different vaccines could be compared against a standard set of decision-making criteria, were not part of its intended use. The evaluation of each vaccine is considered in isolation. In the case of the UNITAG, expert working groups—assigned to each of the vaccine-disease combinations—devised PICO questions and subquestions unique to each vaccine.
The use of PICO questions to guide a systematic review to gather evidence is widely used in, for example, Germany,36 Switzerland,37 the UK38 and USA39 where the availability of new vaccines means they tend to be considered individually. These are also settings where resources for health investments are much higher than LMICs, for example, no benchmark threshold for affordability is used by the Swiss Commission Fédérale pour les Vaccinations (beyond comparison to existing health interventions)37 or the US Advisory Committee on Immunisation Practices39 and the UK broadly follows a figure of £20 000–£30 000/quality-adjusted life-year.38 The consequence for the UNITAG was that the evidence gathered for each of the five vaccines under consideration was tailored to the individual vaccination PICO questions and not immediately comparable between them.
SMART Vaccines 2.0
The SMART Vaccines 2.0 decision-support platform was developed using a multi-citeria decision analysis (MCDA) framework recommended by the US Institute Of Medicine Committee on Identifying and Prioritising New Preventive Vaccines for Development that built on earlier work stemming from the National Vaccine Plan.7 40 The prototype web-based tool was developed at the Fogarty International Center at the NIH and the National Vaccine Programme Office within the US Department of Health and Human Services.6 The tool is designed to facilitate and document the deliberative process of decision-makers to include the different factors driving a decision and their relative weighted importance when considering these factors simultaneously among different vaccine choices.
The SMART Vaccines 2.0 workflow is illustrated in figure 1. Having defined the population, ‘attributes’ or criteria considered important for decision making are identified. These attributes make explicit the rationale for how a vaccine is judged and each criterion is associated with least to most favourable boundaries that apply across each of the vaccines under consideration. In this fashion, the evidence for a vaccine is scored relative to explicit objectives and is directly comparable. Attributes may be qualitative or quantitative and are used to guide what information and evidence needs to be collected for each vaccine considered in an analysis. Attributes themselves are weighted by the decision-makers and this process elucidates any predilections. Evidence is then input and each vaccine is scored against the attribute boundaries. The prioritisation is then the weighted sum of the evidence scored for each attribute.
Workflow for SMART Vaccines 2.0. having identified the population (the region and demographic subpopulation), attributes are identified that describe the dimensions of the prioritisation and these are weighted to determine the relative importance of each attribute. Evidence is collected for each attribute for each vaccine and these, once weighted, are used to form a recommendation. GRADE, Grading of Recommendations Assessment, Development and Evaluation.
Implementation of the decision-support process
Establishing decision criteria
Attributes—the decision criteria—in SMART Vaccines 2.0 were, in this instance, defined to match those of the SIVAC framework. When implemented by the UNITAG, the SIVAC framework, though adaptable, came prepared with four, (implicitly) equally weighted criteria to stimulate discussion in considering the quality of evidence. Those criteria were (1) the severity of the disease (eg, morbidity, mortality and distribution both by age and geography), (2) the benefits of the intervention to recipients (including vaccine safety and effectiveness), (3) economic consideration (primarily the cost–benefit of the vaccine) and (4) programmatic and policy issues (eg, how well a vaccine matches the existing EPI, comparability with coadministration and equity).33
Similar processes to support NITAGs often focus initial deliberations on the balance between measures of disease burden and affordability.41 Recognising the often complicated nature of decision-making, SMART Vaccines 2.0 was designed to broaden the discussion through presentation of multiple domains following a review of relevant stakeholders (figure 2).42 43 These include both quantitative criteria such as the disease burden or cost:benefits, and qualitative issues such as meeting policy directives, promoting equity (in so far as the vaccine will or not address equity or equitable access), or fitting into existing immunisation schedules. Whether these domains are deemed important to the final recommendation or not, it is the discussion triggered by reviewing them that adds richness and critical thinking to the deliberations.
Attributes that can be selected in the SMART Vaccines 2.0 platform that capture the multiple facets of decision making regarding the introduction of vaccines. These attributes extend beyond cost–benefits, but equally they are often difficult to quantify using empirical evidence, but this should not preclude them from considerations and recommendations. During the UNITAG pilot, user-defined attributes were used to match their existing process and these offer greatest flexibility to introduce evidence that might account for unique or tailored criteria, but could equally be used to account for alternative specification of the same variable (ie, to use external data or a different model to estimate costs/benefits). UNITAG, Uganda National Immunisation Technical Advisory Group.
Moreover, the SMART Vaccines 2.0 platform offers the opportunity to define new criteria unique to the decision-makers’ specific circumstances—allowing users to tailor the experience to their local needs and choose how to capture their priorities, whether they become fixed for subsequent recommendations or are reviewed each time.22 See figure 3 for a matrix of the key features in the platform that align with the priority-setting framework described in the introduction.
Alignment between PriorityVax and parameters characterising a successful priority setting framework.
Gathering and inputting evidence
The SIVAC framework was populated with evidence that was gathered through systematic reviews to address PICO questions. Evidence was scored using a GRADE process.35 This process involved a subjective scale to judge the quality of evidence from ‘very low’ to ‘high’ (based on the correspondence between the estimate and true effect sizes). For the UNITAG, these text labels were converted into a numerical score (0, very low to 4, high) and in SMART Vaccines 2.0, attributes were created to mirror those of the framework and boundaries were set to reflect the GRADE scale (ie, 0–4), although SMART Vaccines 2.0 can accommodate qualitative or quantitative evidence from many sources.
Testing sensitivity
Within SMART Vaccines 2.0, alternative scenarios, for example, adjusting the weighting criteria, changing the definitions of benchmarked boundary conditions, inputting new evidence or exploring the uncertainty of existing evidence, can be run quickly. The ability to explore the sensitivity of scenarios can contextualise a score to provide a better understanding of how uncertainty in inputs, assumptions, or implicit biases might influence scoring. This can then help provide credibility to discussions that address uncertainty, and can facilitate alternative ‘what if…’ scenarios to explore different circumstances, for instance, the comparing of funding scenarios.
For this pilot, the evidence was considered fixed, but using SMART Vaccines 2.0 opened a discussion about the relative importance (weights) of the four criteria used in the SIVAC framework. The NITAG determined that the equal weights of the four attributes implicitly built in to the SIVAC framework were not appropriate for the Ugandan context. Therefore, to illustrate the diversity and robustness of UNITAG considerations, the weights given to each criterion were varied in real time. In the first instance, the NITAG came to a consensus about weights through a Delphi scoring approach, second, each member gave weight—separately noting the core (voting) and non-core (non-voting) members. Figure 4 illustrates the variability in both the weights given by members of the NITAG and then the scores of each vaccine scenario. The values of each attribute (eg, the severity of disease) were fixed across NITAG members as this reflects the evidence (agreed on by the Group), whereas the weights are subjective and reflect the inherent biases of individual NITAG members. It was particularly noteworthy that the non-core members, who represented ex officio liaison officers from different stakeholders (eg, the WHO regional office, UNICEF and the Expanded Immunisation Programme) tended to be more varied than the core members.
The weights and scores used by the Uganda NITAG in a pilot of the SMART Vaccines prioritisation tool. Criteria were selected to match the SIVAC framework and scored based on a GRADE-like process. The weight of each criterion was then discussed and presented both as a consensus (grey squares, showing the mean and ±1 SD) of the core (voting) members (blue points) and non-core members (yellow points). The weights from each individual had to add to 100. The variability in the resultant scores, a calculation of weights and scores for the evidence against each criterion, is also shown. Despite variability in the weights, the prioritisation (rank order) of vaccines remained unchanged. it should be noted that these scores formed part, but not the whole of the recommendation that was contextualised based on the available evidence. GRADE, Grading of Recommendations Assessment, Development and Evaluation; NITAG; National Immunisation Technical Advisory Group; SIVAC, Supporting Independent Vaccine Advisory Committee.
Identifying data gaps
The need for comparable data in SMART Vaccines 2.0 also highlighted gaps in knowledge that were somewhat obscured in the SIVAC framework. Because all vaccines are compared on the same basis in SMART Vaccines 2.0, the data need to be directly comparable, (eg, the type of observational data used to measure disease burden needs to be the same). In contrast, by treating each recommendation independently, the SIVAC framework permitted the use of different data types to satisfy component questions, (eg, using suspected cases for one disease and lab-confirmed diagnoses for another, or whether or not the potential for epidemic outbreaks was considered as a subcomponent of the disease burden).
Lessons learnt
Acceptability
In a facilitated postmortem, the use of and results from SMART Vaccines 2.0 were acceptable to the NITAG. The premise satisfied their need to prioritise between vaccines and could validate the conclusions drawn through the SIVAC process. Reporting and data visualisation outputs from SMART Vaccines 2.0 were used to supplement the recommendations made to the MOH.33
Limitations
While values were determined by the NITAG, the hands-on use was facilitated in-person, including leading the NITAG through the interface and discussing the consequences of different choices. This worked for a pilot by demonstrating the potential of the tool, but it did not necessarily demonstrate independent or sustainable use of SMART Vaccines 2.0. Three observations were made that likely apply to all software during adoption of new tools: (1) there is a learning curve which was mollified through facilitation, but in future use it remains unclear how ‘standalone’ SMART Vaccines 2.0 could be, (2) There is a data demand on users. This was raised in earlier versions of SMART Vaccines,7 44 however, much evidence is specific to vaccines in the local context making generic, pre-population of data undesirable. In this pilot, SMART Vaccines 2.0 benefited from the data collection undertaken to populate the SIVAC framework. Generating a similar database is an important and intensive requirement of any ETR process. Ongoing efforts to enhance the capacity of NITAGs to independently gather, interpret and analyse information will likely complement and improve the use of tools like PriorityVax, (3) Advanced users may have their own or existing tools. A new tool therefore may find less traction. For example, formal health technology assessments (HTA) to consider the wider impacts and economics of vaccines require specialised skills and are seldom used by under-resourced NITAGs,45 but are at the core of others.46 It is worth noting that the MCDA framework made accessible through SMART Vaccines 2.0 is increasingly being considered to enhance HTA.47
The Evolution of SMART Vaccines 2.0: PriorityVax
Since this pilot of SMART Vaccines 2.0, the tool has migrated to the Sabin Vaccine Institute. The platform, now named PriorityVax, includes changes to the tool based on the experiences with the UNITAG and additional stakeholder consultation. A major modification focused on deemphasising a built-in compartmental disease/economic model in order to make the tool more generalisable and transparent.
Although the compartmental model was considered a highlight of earlier iterations,44 it has become apparent that users are either unlikely to have the necessary data to run the model (especially across multiple diseases), or they will have their own, preferred, models. A generic model may appear an attractive option given that developing such tools requires specialty skills, but it risks being inappropriate for any given disease if it cannot adequately capture the pertinent disease epidemiology. As a consequence, the emphasis is put on employing the user-defined attributes to construct appropriate attributes that can be populated with available evidence, for example, using a Likert scale to describe the burden of disease. This additional flexibility also permits competition of non-vaccine interventions (eg, screening or treatment) within the tools framework, allowing for a more holistic consideration of health system constraints and benefits. Because MCDA is based on the combination of weights and values (evidence) both of which can be entered by the user rather than generated by the disease/economic model, this simplifies the workflow considerably (figure 1). That said, some aspects remain constrained by the structure of the underlying code.
Simplifying the workflow has also meant that the process of using PriorityVax is now clearer and more user-friendly. It is envisaged that the updated tool will be easier to support remotely if required, including as part of ETR training (eg, encouraging exploration of sensitivity to bias and data uncertainty). The intention is that PriorityVax becomes sufficiently self-explanatory such that facilitation is at a minimum or unnecessary, thereby giving greater ownership and independence to the country-level users and ensuring the sustainability of the tool. Gathering and evaluating evidence within an ETR framework will always require considerable effort and in the interests of time, the UNITAG prioritisation process was facilitated by the SMART Vaccine 2.0 team. Continued development of supporting ETR guidance resources combined with the implementation of the tool is envisaged to allow country-level users to make their own choices that reflect their unique circumstances without the need for externally guided inputs.
The last significant change has been to move the modelling code into C#, an accessible and open source language. This has improved the stability and performance of the website and will permit easier implementation of future updates.
Conclusions
In resource-constrained settings, prioritising among new and underused vaccination strategies or selecting between vaccine products, will support long-term commitment and sustained investments. A local decision of which or whether, to introduce a vaccine is a fundamental component of developing evidence-informed recommendations given the expanding range of vaccines available and increasing costs implied. PriorityVax (the successor to SMART Vaccines 2.0) is a platform designed to capture deliberations around prioritisation, to explicitly evidence the importance of different criteria and produce a consistent documented process that supports transparent EIDM.48 In the piloted implementation, the platform’s value was demonstrated by its acceptance among the users to support a real-world priority-setting process.
Acknowledgments
We are grateful to the Uganda NITAG members and secretariat for their enthusiastic engagement and partnership in piloting SMART Vaccines 2.0. We are appreciative of the collaborative engagement offered by Antoinette Ba-Nguz who represented the SIVAC initiative. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the US National Institutes of Health or Department of Health and Human Services or the UNITAG.
References
Footnotes
Handling editor Seye Abimbola
Contributors BJJM and SLK drafted the manuscript following piloting of SMART Vaccines 2.0 collaboratively with the Uganda NITAG, which included PW, CN and NKS who also gave invaluable assessments reflected in manuscript development. All authors read, edited and approved the manuscript.
Funding This work was supported by the Fogarty International Center, National Institutes of Health, USA.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement There are no data in this work.