Article Text
PDF
Abstract
Introduction Evidence-to-decision (EtD) frameworks intend to ensure that all criteria of relevance to a health decision are systematically considered. This paper, part of a series commissioned by the WHO, reports on the development of an EtD framework that is rooted in WHO norms and values, reflective of the changing global health landscape, and suitable for a range of interventions and complexity features. We also sought to assess the value of this framework to decision-makers at global and national levels, and to facilitate uptake through suggestions on how to prioritise criteria and methods to collect evidence.
Methods In an iterative, principles-based approach, we developed the framework structure from WHO norms and values. Preliminary criteria were derived from key documents and supplemented with comprehensive subcriteria obtained through an overview of systematic reviews of criteria employed in health decision-making. We assessed to what extent the framework can accommodate features of complexity, and conducted key informant interviews among WHO guideline developers. Suggestions on methods were drawn from the literature and expert consultation.
Results The new WHO-INTEGRATE (INTEGRATe Evidence) framework comprises six substantive criteria—balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations—and the meta-criterion quality of evidence. It is intended to facilitate a structured process of reflection and discussion in a problem-specific and context-specific manner from the start of a guideline development or other health decision-making process. For each criterion, the framework offers a definition, subcriteria and example questions; it also suggests relevant primary research and evidence synthesis methods and approaches to assessing quality of evidence.
Conclusion The framework is deliberately labelled version 1.0. We expect further modifications based on focus group discussions in four countries, example applications and input across concerned disciplines.
- public health
- health systems
- health policy
This is an open access article distributed under the terms of the Creative Commons Attribution-Non commercial IGO License (CC BY 3.0 IGO), which permits use, distribution,and reproduction for non-commercial purposes in any medium, provided the original work is properly cited. In any reproduction of this article there should not be any suggestion that WHO or this article endorse any specific organization or products. The use of the WHO logo is not permitted. This notice should be preserved along with the article’s original URL.
Statistics from Altmetric.com
Key questions
What is already known?
Evidence-to-decision (EtD) frameworks help to ensure that all criteria of relevance in a given guideline development or other health decision-making process are considered in a systematic way.
What are the new findings?
The WHO-INTEGRATE (INTEGRATe Evidence) framework is a new EtD framework that is rooted in the norms and values of the WHO, which are agreed on by all WHO Member States.
The framework was developed to be applicable to all health interventions, although it is particularly well suited for decisions about population-level and system-level interventions at the global as well as national levels.
The WHO-INTEGRATE framework offers structured definitions for each of the six substantive criteria as well as the meta-criterion quality of evidence; example questions and suggested methods are provided to facilitate uptake.
What do the new findings imply?
As part of a more holistic approach, the framework is devised as a tool to facilitate structured reflection and discussion from the beginning to the completion of a guideline development or other health decision-making process; this entails prioritisation among criteria and subcriteria to ensure appropriate evidence collection and appraisal.
Background
Health decision-making at local, national, regional and global levels is complex,1–3 and can be influenced by a broad range of factors.4–9 Their importance varies depending on the type of health decision and the decision-making context,10 11 where context can relate to the institutional context (eg, Ministry of Health vs municipality), as well as the broader physical and social context, including epidemiological, geographical, sociocultural, political and other aspects.12 Health decision or evidence-to-decision (EtD) frameworks are intended to ensure that all important factors—in the form of decision criteria—are considered in a systematic and transparent way.13–19 They provide a structured approach for guideline panels or other decision-making bodies to consider the available evidence and to make informed judgements about the advantages and drawbacks of a given health decision; this approach can comprise substantive criteria as well as procedural aspects. Health decision frameworks have been applied in a variety of decision-making contexts.20 21
Guidelines by the WHO provide recommendations for clinical practice, public health and health system strengthening, and are intended to support health decision-makers in prioritising among or selecting suitable clinical, public health or health system interventions. When formulating recommendations, WHO generally uses an EtD framework which encompasses eight criteria: quality of evidence (in relation to intervention benefits and harms), values and preferences (in relation to outcomes), balance of benefits and harms, resource implications, priority of the problem, equity and human rights, acceptability, and feasibility (table 10.1 of the WHO Handbook for Guideline Development).22
Chapter 10 in the WHO Handbook for Guideline Development 22 was written by one of the lead authors of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) and the GRADE EtD frameworks. The criteria in the current WHO EtD framework represent an advanced but—given their publication in 2014—not the final version of the GRADE EtD framework, which offers different versions for clinical recommendations from an individual or population perspective, coverage decisions, health system/public health decisions and recommendations about tests.13 23 In a recent systematic review of frameworks concerned with generic health decision-making and resource allocation processes, health technology assessments, as well as very specific health decisions, the GRADE EtD framework emerged as the best fit-for-purpose framework (Stratil et al, forthcoming). In particular, this framework can be applied across diverse types of health decisions and was developed following an iterative and multipronged process, combining a literature review, brainstorming and feedback from stakeholders,24 with application of the framework to examples and user-testing.20 25
However, a number of weaknesses were identified with the GRADE EtD frameworks (Stratil et al, forthcoming). First, the framework was developed using a pragmatic approach and lacks an explicit theoretical or conceptual basis. This makes it difficult to assess objectively whether the set of criteria is complete and organised in a meaningful way.
Second, while the frameworks are largely congruent with WHO norms and values, they do not sufficiently consider the central role of the social and economic determinants of health26 and the implications of health sector or intersectoral interventions for society as a whole. This is particularly important given the significant shifts in the global health landscape and the objectives and values manifest in the Sustainable Development Goals (SDGs),27 which are likely to shape health decision-making in the future.
A third concern is whether the decision criteria in the GRADE EtD framework are sufficiently complete and useful for decisions about complex interventions and/or the complex systems in which these are implemented, especially interventions aiming to bring about system-level changes.28
Fourth, the frameworks were originally developed in consultation with healthcare decision-makers in Europe, Canada and Africa, the majority of whom were physicians with significant clinical experience and research training.24 As a result, the frameworks may not be entirely suitable to broader public health and health system decision-making contexts, particularly in low-income and middle-income countries of Asia and Latin America.
A final and important concern relates to consistency in the application of the GRADE EtD frameworks within the WHO guideline development processes. While there are exemplar guidelines, where the WHO EtD framework has been employed as intended,29 30 many WHO guideline development groups focus extensively on the criterion balance of benefits and harms and apply the remaining criteria as a check box exercise rather than as a process that structures the development of guidelines from the start: from scoping a guideline and prioritising questions, to collecting, synthesising and appraising evidence, to formulating recommendations (SL Norris, 2017, personal communication). While there are many potential reasons for this, the current content and structure of the GRADE EtD framework may result in superficial use rather than indepth collection and assessment of evidence for the different criteria. In particular, guidance on how to frame questions for and collect evidence towards criteria beyond balance of health benefits and harms appears to be missing.
This paper, one of a series exploring the implications of complexity in systematic reviews and guideline development, reports on the development of a new EtD framework that is rooted in WHO norms and values and suitable for a broad range of health interventions, including complex interventions and interventions delivered in complex systems.
The paper addresses the following three objectives:
Develop an EtD framework that (a) is firmly rooted in WHO norms and values and reflective of the changing global health landscape, and (b) encompasses a comprehensive set of criteria suitable for clinical practice, public health and health system interventions.
Explore the value of this framework in relation to (a) complexity in individual-level as well as population-level and system-level interventions, (b) the views of developers of the WHO guidelines (global level), and (c) the views of users of the WHO guidelines (national level).
Facilitate uptake of the framework by emphasising the need for structured, evidence-based reflection and suggesting methods to populate the criteria with evidence in the context of decision-making under uncertainty.
The EtD framework developed out of this process is referred to as the WHO-INTEGRATE (INTEGRATe Evidence) framework version 1.0. It is proposed for use in the WHO guideline development as well as in other guideline development or health decision-making processes at the global or national level. It is intended to be used holistically—from the beginning of a health decision-making process to formulating recommendations or making a decision at the end of this process.
Methods
In addressing these objectives, we followed a three-step approach, as illustrated in figure 1.
This paper provides an overview of the research project with all of its constituent components. It presents the current version of the framework (WHO-INTEGRATE framework version 1.0) and its intended use. It also reports the detailed methods and findings for steps 1a, 2a and 3, as well as an overview of the methods and findings for steps 1b and 2b. A full account of the methods and findings of step 1b is currently in preparation (Stratil et al, forthcoming). An integrated analysis of the views of those developing (step 2b) and using WHO guidelines (step 2c) with respect to the WHO-INTEGRATE framework will also be published separately (Stratil et al, forthcoming).
Towards a useful and operational WHO-INTEGRATE (INTEGRATe Evidence) framework.
Step 1: Development of the framework
In step 1a we analysed WHO norms and values and, rooted in these norms and values, proposed a structure for the WHO-INTEGRATE framework and derived preliminary criteria. A universally agreed normative theory for health does not exist, but most rivalling theories converge on a set of principles.31 As the use of these principles is less restrictive than the choice of one theory over another, we pursued a principles-based approach,31 32 and used WHO norms and values as the guiding principles for developing a new EtD framework. Given the complexities of normative orientation in modern pluralistic and globalised societies, we believe that WHO norms and values represent a useful foundation: they are rooted in the universally recognised concept of human rights and receive their legitimacy from having been agreed on by all 194 Member States of the WHO. To identify WHO norms and values of relevance to the process of guideline development and implementation, we used the WHO Constitution33 and chapter 5 ‘Incorporating equity, human rights, gender and social determinants into guidelines’ of the WHO Handbook for Guideline Development22 as a starting point. Given the emphasis in these two documents on human rights, equity and non-discrimination, social determinants of health and the role of health systems, we retrieved and analysed relevant related documents,34–40 including several public health ethics frameworks.16 18 41–50 We also reviewed the SDGs51 in view of their likely impact at global and national levels and as WHO is mainstreaming these throughout the organisation’s work.52–54
From these documents and sources, we derived principles and concepts. The structure of the WHO-INTEGRATE framework was developed via an iterative process among coauthors. We explored the meaning of different principles and concepts and assessed overlap and redundancies, making rearrangements to derive preliminary criteria. In doing so, we used a structure and wording as close as possible to the existing GRADE EtD framework to build on its strengths and to maximise potential synergies. On several occasions, we also consulted with members of the WHO Guidelines Review Committee as well as other WHO staff considered experts on selected principles or concepts (see Acknowledgements).
During the development process, we focused on substantive criteria or what decisions are based on (eg, cost, acceptability) rather than procedural criteria or how the decision-making process is conducted (eg, composition of guideline panels, participation, transparency). This is consistent with the approach promoted by the WHO Handbook for Guideline Development,22 whose overall purpose is to specify procedural rules for an objective, transparent and acceptable guideline development process. Embedded in these procedural rules, the current WHO EtD framework (table 10.1 of the WHO Handbook for Guideline Development)22—and, by extension, the WHO-INTEGRATE framework presented here—is concerned with how to facilitate the use of evidence in decision-making in a structured and comprehensive manner. It is important to note that a distinction between structural and procedural aspects is widely practised in guideline development and several other health decision-making processes,5 6 19 55 but is not commonly seen in the public health ethics literature.43 44 46 56
In step 1b we refined the preliminary criteria derived from WHO and other related documents and supplemented them with a comprehensive set of subcriteria; we also developed definitions for criteria and example questions relating to each of the subcriteria. We conducted an overview of systematic reviews of criteria used in decision-making, priority setting and resource allocation processes for health to derive a comprehensive set of health-relevant criteria (Stratil et al, forthcoming). We then compared the preliminary criteria developed in step 1a against this comprehensive set of criteria and subcriteria. To do so, one author (JMS), in a discussion with a second author (EAR), allotted the subcriteria obtained from the overview of systematic reviews to the preliminary criteria within the WHO-INTEGRATE framework. Subcriteria that did not fit were kept in a separate category. Any uncertainties were resolved in discussion with a third author (RB).
We then prepared definitions for each of the criteria using the above-described source documents for health norms and values, existing health decision frameworks (Stratil et al, forthcoming), and any definitions or descriptions provided in the publications included in our overview of systematic reviews of criteria (Stratil et al, forthcoming). Where appropriate, we also drew on additional key documents (eg, Scott et al 57 for the definition of acceptability, Hultcrantz et al 58 for the conceptualisation of quality of evidence, and Maeckelberghe and Schröder-Bäck59 for details on the subcriteria for human rights and sociocultural acceptability and health equity, equality and non-discrimination). Each definition (1) provides an overall definition of the criterion, (2) offers details and explanations regarding the subcriteria, and (3) gives guidance on how the criterion in question influences the recommendation.
As we prepared definitions, we also examined the extent to which the criteria and subcriteria relate to the intervention itself versus the health system and the broader context, in which an intervention is implemented. For example, the same label (eg, equity) may be employed to describe different underlying concepts, relating to process versus outcome (an intervention can either be implemented taking equity principles into account, or it can increase or decrease equity in health outcomes) and the point in time when these criteria apply (eg, equity before, during or after intervention implementation). To enable better access to sometimes abstract constructs, we also developed example questions for each of the subcriteria, drawing on the same set of documents as above.
Step 2: assessment of framework value
In step 2a we explored whether the WHO-INTEGRATE framework would be able to accommodate different types of health interventions and different features of complexity. We assessed to what extent the WHO-INTEGRATE framework would be able to accommodate features of distinct types of health interventions.60 We broadly distinguished between interventions targeting individuals (eg, diagnosis, treatment or preventative measures addressed at individuals), interventions targeting populations, and interventions targeting the health system or other systems. Population-level interventions encompass those concerned with whole populations or population groups as defined by their age, sex, risk factor profile or other characteristics; they are often implemented in specific settings or organisations (eg, school health programmes). System-level interventions specifically redesign the context in which health-relevant behaviours occur; they are often implemented through geographical jurisdictions from national to local levels (eg, laws and regulations regarding the taxation, sale and use of tobacco products). Health system interventions represent a specific type of system-level intervention and often result in complex rearrangements across multiple health system building blocks (eg, task shifting as a process of delegating specific health service tasks from medical doctors or nurses to less specialised health workers). Interventions implemented at any of these levels can be conceptualised and analysed from a complexity perspective. To do so, we mapped core and additional components of complex interventions as defined in the iCAT_SR tool61 and sources of complexity in systems reported in another paper in this series28 against the included criteria.
In step 2b we examined the usefulness and relevance of the WHO-INTEGRATE framework and its criteria among those developing WHO guidelines. We conducted key informant interviews with individuals who had recently participated in a WHO guideline development process. In consultation with the Secretariat of the WHO Guidelines Review Committee, we purposively selected three ongoing or completed guidelines that had applied the current WHO EtD framework,29 62 63 seeking to cover distinct types of health interventions and positive as well as more difficult experiences with the application of the framework. For each guideline, we interviewed the WHO staff coordinating the guideline, the Chair of the guideline development group and the methodologist. The interviews were semistructured and used a pretested interview guide concerned with practical considerations (eg, understandability, operationalisability), as well as an assessment of missing and redundant criteria of the WHO-INTEGRATE framework. Interviewees were also asked to reflect on the implications of the WHO-INTEGRATE framework for evidence collection and guideline formulation. Interviews were held between June and November 2017 either face-to-face at the WHO Headquarters in Geneva or by telephone (JMS). Interviews were audiotaped and transcribed; data were then analysed by two researchers (JMS and IBS) using qualitative content analysis.64 We employed a combination of deductive (based on the guiding research questions) and inductive approaches using the software MAXQDA (VERBI Software, Berlin).
Step 3: facilitation of framework uptake
We critically examined how to enable use of the WHO-INTEGRATE framework as intended, and generated a table linking the criteria with suggested methods for primary research, evidence synthesis and assessing quality of evidence. The current WHO EtD framework is intended to be used right from the planning stages of a guideline, to help derive relevant questions and structure the process, but in practice it is usually used at the end of a guideline process to help decide on the recommendations. To determine how the new framework could be used more holistically, we reflected on the literature reviewed in the context of developing the WHO-INTEGRATE framework and sought feedback from a large number of experts (see Acknowledgements). We specifically sought suggestions on how to use the framework during the early stages of the guideline development process and in a context-specific manner.
To make it easier for guideline panels to populate the criteria in the framework with evidence, we identified types of primary research, evidence synthesis methods and methods for assessing evidence quality that could inform each criterion. To accomplish this, two researchers (AP and EAR) reviewed the research questions and methods described or mentioned in the systematic review of health decision frameworks and the overview of systematic reviews (Stratil et al, forthcoming). We also consulted a broad range of experts comprising other authors of papers in this series, selected guideline development organisations (eg, Guidelines International Network, UK National Institute for Health and Care Excellence) and researchers with an interest in evaluating complex health technologies (see Acknowledgements).
Results
Developing the preliminary framework
Using the review of the WHO Constitution,33 chapter 5 of the WHO Handbook for Guideline Development22 and other source documents, we identified six major, partly overlapping concepts. Further sorting of these yielded four sets of principles and concepts (human rights principles, ethical principles, sustainability elements and health system goals and building blocks). Figure 2 illustrates how we derived preliminary criteria from WHO norms and values.
Sources and concepts for deriving principles-based preliminary criteria rooted in WHO norms and values.
Human rights principles, for the purposes of this framework, were primarily derived from international human rights law and its interpretation by the Committee on Economic, Social and Cultural Rights’ General Comment on the Right to the Highest Attainable Standard of Health (Art 12).34 These contain the interrelated concepts of availability and accessibility of public health and healthcare facilities, goods and services, which are required to be of appropriate quality and acceptable to users. They also include the general human rights principles of equity and non-discrimination, accountability and participation.
Given the large number of biomedical and public health ethics frameworks,44 46 56 in consultation with WHO, we structured the ethical principles primarily according to the public health ethics framework of Childress and colleagues. This framework inter alia includes the aspects of producing benefits, avoiding harms, maximising the balance between benefits and harms, as well as distributive justice and autonomy.41 Based on analytical tools by the Nuffield Council of Bioethics,45 we also added the principle of low intrusiveness, which is related to privacy and dignity.
Acknowledging the importance of the social determinants of health and the SDGs, we derived sustainability elements to capture the wide range of factors that promote conditions in which people can lead a healthy life and allow societies and individuals to develop and flourish; these sustainability elements also reflect the societal impact that interventions can have beyond health outcomes. Importantly, good health is both a precondition for achieving sustainable development and an outcome of sustainable development.65
To capture the importance of feasibility of implementation as well as the impact of interventions on the health system, we used the WHO health systems framework with its four goals (ie, improved health, responsiveness, social and financial protection, improved efficiency) and six building blocks (ie, leadership and governance, financing, medical products, vaccines and technologies, information, health workforce, service delivery infrastructure).19 38–40
Figure 3 presents the WHO-INTEGRATE framework with its six criteria: balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations. A seventh criterion, quality of evidence, represents a metacriterion that applies to each of the six substantive criteria. All seven criteria are relevant to health decision-making and the formulation of recommendations as part of the guideline development process. Each criterion may apply to interventions targeting individuals, populations or systems, or any combination of these levels.
The WHO-INTEGRATE (INTEGRATe Evidence) framework version 1.0.
While priority of the problem featured in both the health decision frameworks included in our systematic review (eg, Alonso-Coello et al 13) and in the overview of systematic reviews of criteria (eg, Guindo et al 6), we did not include this as a stand-alone substantive criterion for two reasons: First, many of the aspects included, for example, political will or public concern, are used to inform the decision to develop a guideline (or make another health decision) and thus apply before the start of the guideline development process. Second, selected aspects are captured under the other six substantive criteria, for example, burden of disease features under balance of health benefits and harms, and large cost of disease to health system features under financial and economic considerations.
Defining criteria, subcriteria and example questions
Our overview of systematic reviews yielded more than 30 systematic reviews that contained several thousand criteria and subcriteria currently used in decision-making (Stratil et al, forthcoming). Recurrent aspects addressed by the subcriteria focused on the health outcomes and benefits of the intervention, health benefit for individuals and the benefit for society as a whole, the societal importance of the disease, economic considerations, quality or uncertainty of evidence, as well as population priorities, priorities within the health system and stakeholders’ interests and pressures. Feasibility criteria were concerned with the available budget, the capacities within the health system, technological complexity and acceptability of the intervention within society. Some systematic reviews were primarily concerned with interventions that would benefit vulnerable or marginalised populations (eg, children, mothers, people with lower socioeconomic status). In many reviews, normative criteria such as ethics, justice or fairness were mentioned without clear definitions or contextualisation. This comprehensive list did not yield any further criteria beyond the seven presented in figure 2. It did, however, provide many subcriteria as well as elements used in the development of detailed definitions and example questions for each criterion.
Suitability of the framework for decisions about complex health interventions
An earlier paper in this series28 emphasises the importance and added value of reviewing evidence from a complex systems perspective. In developing the new EtD framework, we wanted to ensure that it would be fit for purpose when making decisions about complex interventions implemented in complex systems. We first explored to what extent different features of intervention and system complexity apply to two broad categories of interventions, that is, individual-level versus population-level and system-level interventions (table 1). Notably, even population-level and system-level interventions (eg, regulations and programmes to increase access to improved sanitation) eventually bring about changes in individual behaviour (eg, use and maintenance of toilets or latrines). Some criteria apply to a greater extent with population-level and system-level interventions (eg, societal implications) than individual-level interventions. Some subcriteria may take on a different meaning when applied to individual-level versus population-level and system-level interventions (eg, autonomy). Broadly speaking, most features of complex interventions apply to both individual-level and population-level/system-level interventions but are more salient for the latter. In contrast, many features of complex systems only apply to population-level and system-level interventions.
The last column of table 1 illustrates that distinct features of complexity do not neatly map onto specific criteria. Instead, distinct features of complexity usually affect multiple, sometimes all, criteria in the WHO-INTEGRATE framework. For example, the worked example of childhood obesity, introduced in an earlier paper in this series,28 discussed adaptivity of the system in response to raised taxes on soft drinks (eg, creation of lower-sugar alternatives by the soft drinks industry). This adaptivity can thus influence the balance of health benefits and harms (eg, consumption patterns of soft drinks change but in less pronounced ways, thereby dampening the expected effect on childhood obesity), and it may even have unwanted social consequences by stigmatising those unable to afford soft drinks (social impact). Raising taxes on only one sugar-sweetened product may lead to increasing the sugar content of other sugar-sweetened products (impact on economy, broader positive or negative health-related impacts) or have implications on agricultural production patterns nationally and internationally (impact on economy and environmental impact), illustrating the complexity of downstream implications of a ‘simple’ intervention. Drawing on the same worked example, box 1 illustrates how a simple linear perspective on the effect of an intervention will place the emphasis on one or a few criteria for decision-making, whereas a complexity perspective may take all criteria into account when making a recommendation.
Thinking through the criteria in relation to raised taxes on soft drinks and their implications
A simple perspective on raising taxes on soft drinks would emphasise the linear impacts of this intervention on consumption of sugar-sweetened beverages (intermediate outcome) and different measures of childhood obesity (ultimate outcome of interest); with this perspective, the criterion balance of health benefits and harms would warrant the most attention. A complexity perspective on the same intervention would not start off with a preconception about a single criterion being most influential but carefully examine all criteria. For illustration purposes, this complexity perspective would examine acceptability among and likely reactions from different groups of stakeholders (eg, children, their parents), and pay specific attention to the response from vendors and producers of soft drinks (eg, potential sugar reduction in drinks with implications for the prices of these drinks), which may dampen the expected effect of the tax intervention, in terms of changes in consumption patterns, perceptions of the intervention and changes in social norms.
This complexity perspective would also encompass potential negative impacts on health equity, equality and non-discrimination (eg, expected or unexpected changes in consumption patterns across different socioeconomic or other population groups), explore positive or negative social, environmental or economic impacts (eg, changes in social norms in relation to sugar-sweetened beverages or their alternatives being more or less desirable among different population groups, changes in acceptability of further interventions to reduce sugar consumption), adopt a societal perspective in estimating the financial and economic impacts of the intervention (eg, including how costs and benefits of the raised taxes are distributed among different stakeholders and sectors), and pay attention to feasibility and health system considerations (eg, implications for human resources involved with other ongoing efforts to reduce consumption of sugar-sweetened beverages and childhood obesity).
Usefulness of the framework from the perspective of WHO guideline developers
The key informants we interviewed had been involved in developing three very different guidelines—the WHO recommendations on antenatal care,29 the WHO consolidated guideline on sexual and reproductive health and rights of women living with HIV,62 and the WHO guideline on risk communication (online supplementary table S1).63 Each of these guidelines faced different challenges in terms of scope, availability of evidence and ability to incorporate multiple perspectives. All three had used the current WHO EtD framework with varying success. The diverse experiences and viewpoints of the key informants on the practical application of these criteria in guideline development were helpful in refining the framework. Further detail on and complete findings from the key informant interviews will be reported separately (Stratil et al, forthcoming).
Supplementary file 1
Most participants commented positively on the WHO-INTEGRATE framework and highlighted the value of a criterion assessing societal implications, as well as the broader and more detailed specification of the criteria human rights and sociocultural acceptability and health equity, equality and non-discrimination. Two participants questioned the added value of the new EtD framework, since any guideline development process led by an experienced methodologist would automatically address the details covered in the subcriteria. Several participants were concerned about the workload that the use of the WHO-INTEGRATE framework might add to the guideline development process.
Specific remarks were made in regard to (1) missing criteria and subcriteria; (2) the hierarchy and order of criteria and subcriteria; (3) overlap and redundancies between criteria and subcriteria; (4) the precise wording and definitions of criteria; (5) the need for (more) guidance on how to use and interpret criteria and subcriteria; (6) the challenges of identifying and synthesising the required evidence; (7) resource, time and skill implications for the guideline development process; as well as (8) procedural aspects for using the framework in the guideline development process.
In response to these concerns and suggestions, we made several modifications, including changing the name and definition of several criteria and subcriteria to improve clarity and reduce overlap. We also expanded the example questions for the subcriteria to improve understandability and facilitate the development of specific questions for a given guideline. Moreover, we added suggestions on how to prioritise among criteria and subcriteria in a problem-specific and context-specific manner. Finally, we emphasised the importance of incorporating the voices of those directly affected by the recommendations into the guideline development process.
Table 2 presents the WHO-INTEGRATE framework version 1.0 criteria with abbreviated definitions and lists subcriteria. Online supplementary table S2 provides detailed definitions of the criteria as well as example questions for each of the sub criteria.
WHO-INTEGRATE framework version 1.0: criteria with abbreviated definitions, subcriteria and implications for a recommendation. All criteria are relevant for all interventions in health decision or guideline development processes. For subcriteria there should be a discussion as to which are most relevant and if or how evidence should be collected to inform these. Online supplementary table S2 provides detailed definitions of the criteria and example questions for each of the subcriteria.
Facilitating uptake: using the framework holistically and populating the criteria with evidence
The WHO-INTEGRATE framework is intended to improve transparency in health decision-making by supporting a structured process of reflection and discussion in a problem-specific and context-specific manner. To be most effective, this process must begin at the start of a guideline or other health decision-making process and must take evidence into account. The WHO-INTEGRATE framework is not intended as a ‘tick-box exercise’; there must be prioritisation of the most relevant criteria and subcriteria depending on the questions addressed by a given guideline, and the time and resources at disposition. It would be impossible and probably unnecessary for every guideline development or health decision-making process to examine all subcriteria. This flexibility can, however, lead to misuse, as stakeholders may disproportionately (eg, academics from high-income countries) or unduly (eg, participants with substantial declared or undeclared financial or other conflicts of interest) influence the decision-making process. Safeguards can be put in place through explicit procedures, in particular in relation to the composition of guideline panels or other decision-making groups. The WHO-INTEGRATE framework is also not an algorithm for integrating evidence across different criteria: making decisions under uncertainty and agreeing on trade-offs across criteria and subcriteria and among (and within) diverse stakeholder groups remain a core task for a guideline panel.
All criteria are important and should be reflected on, but their relevance varies depending on the type of health decision and the decision-making context. In contrast, not all subcriteria are always relevant. At the start of a guideline or other decision-making process, an appropriately composed guideline panel or other decision-making group needs to discuss which of the subcriteria are applicable and useful in relation to the nature and specific characteristics of the intervention (see table 1); this group will also need to consider the specific information needed to populate criteria or subcriteria (see table 3). Complexity in the intervention and complexity in the system into which this intervention is implemented can usually be detected; the critical question is whether it is of value to examine this complexity in depth (see box 1 in this paper and box 2 in an earlier paper in this series28). This prioritisation process should take the views of relevant stakeholder groups into account; which stakeholder groups are relevant depends on the nature of the problem and the institutional as well as broader physical and social context. In principle, these should include those directly affected by the intervention (eg, patients, beneficiaries), those financing (eg, health insurance providers, ministries of health, other ministries) or implementing the intervention (eg, healthcare providers, public health professionals, professionals outside of the health sector), as well as the general public.
WHO-INTEGRATE framework version 1.0: criteria and suggested types of primary studies, evidence synthesis methods and approaches to assessing quality of evidence
A systematic weakness in many guideline development and other health decision-making processes is that consumer participation is obviated and guideline panels often substitute their own values and views for those of patients/beneficiaries. The voices of patients/beneficiaries and other relevant stakeholder groups can be incorporated through direct participation or representative surveys66 as well as qualitative research (see table 3).
The guideline panel will also need to decide how best to populate the criteria with evidence and whether a formal evidence synthesis or a more pragmatic approach is warranted for each. This decision will be influenced by the relevance of criteria and subcriteria in relation to a specific intervention or decision, and by the likely types and quantity of evidence available, as well as time and resource constraints. At the end of the process, the guideline panel will need to reassess the criteria and relevant subcriteria in light of the assembled evidence and make a judgement regarding each criterion.
Table 3 suggests relevant types of primary research, evidence synthesis or mapping methods, streamlined or pragmatic approaches, as well as methods to assess the quality of evidence for each of the six substantive criteria. We provide a collection of suitable primary research and synthesis approaches, but make no firm distinction between more or less suitable methods. We note that the approach to gathering evidence may depend on the criterion: for some criteria a systematic review will be most appropriate, while for others a representative survey or other single primary study may be more suitable. Surprisingly, the majority of the health decision frameworks included in our systematic review (Stratil et al, forthcoming) did not offer insights for operationalising frameworks, for example by specifying research questions or suggesting methods for primary research or evidence synthesis. The GRADE EtD framework13 67 and the EUnetHTA (EUropean network for Health Technology Assessment) core model68 provided some methods. We also identified relevant information in the following sources: the EVIDEM (Evidence and Value: Impact on DEcisionMaking) framework,14 Marckmann and colleagues,16 the health systems framework19 and publications included in our overview of systematic reviews of criteria.69 Expert consultation played a critical role in identifying methods for inclusion in table 3.
Discussion
Added value of the WHO-INTEGRATE framework
The WHO-INTEGRATE framework represents a new comprehensive EtD framework that is rooted in WHO norms and values. It offers an explicit conceptualisation of each criterion and a rationale for including relevant concepts as criteria or subcriteria. The WHO norms and values apply across all WHO Member States and settings, and the new framework should, in principle, be relevant for health decision-making at global, national and subnational levels. It reflects a broad understanding of health and its determinants and takes account of complex interventions and complex systems perspectives. It emphasises sustainability and the interconnectedness between health and other sectors, inherent in the SDGs. While the framework is conceived for individual-level, population-level and system-level interventions, it is likely to be particularly well suited for public health and health system interventions characterised by complexity and/or approached from a complexity perspective. The WHO-INTEGRATE framework is intended as a tool to facilitate structured reflection and discussions from the beginning of a guideline development or other health decision-making process. This has ramifications in terms of the need to prioritise among criteria and subcriteria and the need to collect evidence for each. The framework supports this process by offering structured definitions for each criterion and example questions for each subcriterion, and by suggesting methods for primary research, evidence synthesis and assessing the quality of the evidence.
There are many similarities between the WHO-INTEGRATE framework and the widely used GRADE EtD framework. As stated in our methods, we deliberately attempted to stay as close as possible to the GRADE EtD framework, thus building on established terms and concepts (eg, balance of health benefits and harms). In contrast, criteria with a strong normative foundation (eg, health equity, equality and non-discrimination) were much less developed in the GRADE EtD framework; notably, the criterion societal implications, which has its roots in the recognition of the multisectoral determinants of health, is absent from the GRADE EtD framework. There are also more fundamental differences. While the GRADE EtD framework emphasises the efficacy/effectiveness of interventions and their potential harmful impacts, there is no inherent weighting of criteria in the WHO-INTEGRATE framework: guideline panels must decide in a context-specific and problem-specific manner which criteria and subcriteria are most relevant. Moreover, in contrast to the narrower certainty of evidence concept in the GRADE EtD framework, the WHO-INTEGRATE framework has deliberately adopted a broad quality of evidence concept that applies across all criteria and is not linked to a prespecified grading system. For several criteria (and/or subcriteria) GRADE70 71 and GRADE CERQual (Confidence in the Evidence from Reviews of Qualitative Research72) are the most appropriate approaches to examining quality of evidence, and we would encourage users of the WHO-INTEGRATE framework to adopt these. In fact, another paper in this series explores how complexity can be considered when assessing the certainty of evidence on intervention effectiveness.73 For other criteria (and/or subcriteria), these existing tools are not well suited, and we hope that more appropriate approaches will become available—whether through further developments within the GRADE Working Group or independent efforts.
The GRADE EtD framework allows for tailoring of criteria, for example by considering a detailed judgement as a stand-alone criterion or by removing a criterion from the GRADE EtD framework and considering it prior to the start of the decision-making process13; in fact, refinement of the GRADE EtD framework continues and has already resulted in suggestions towards more detailed specifications of selected criteria.74 Similarly, we expect various developments towards a version 2.0 of the WHO-INTEGRATE framework (see below). We thus envisage specific innovations to be adopted across these evolving frameworks and, potentially, convergence over time.
Strengths and limitations of the development process
In developing the WHO-INTEGRATE framework, we combined a principles-based approach with an overview of systematic reviews of decision criteria and thus ensured a solid, comprehensive normative foundation. We were explicit and transparent as to how criteria (see figure 1) and subcriteria (Stratil et al, forthcoming) were derived. While there is some conceptual overlap at the level of the criteria (eg, societal implications and financial and economic considerations), there are no significant redundancies among the subcriteria (Stratil et al, forthcoming). Cross-linkages among the criteria are emphasised in the definitions and example questions.
Solely adapting the substantive criteria may be insufficient to overcome limitations in guideline development or other decision-making processes.22 The WHO-INTEGRATE framework is concerned with substantive criteria; it does not comprise procedural criteria but is intended to be embedded in a clearly specified health decision-making process as described, for example, in the WHO Handbook for Guideline Development.22 We recognise that transparent and inclusive procedures are essential to achieve legitimate health decisions and to resolve reasonable disagreement based on competing criteria and the various individual, social, cultural and political values affecting their interpretation and the explicit or implicit weight assigned to them. In this context legitimacy refers to the reasonableness, or acceptability, of decisions as perceived by the population.75 76 Compromised legitimacy may hinder the effective implementation of guidelines or other health-relevant decisions. Transparent and inclusive procedures require, among other considerations, the involvement of relevant stakeholders in the decision-making process, the public announcement of forthcoming decisions including their underlying argumentation, and the instalment of mechanisms for appeal.75 76 This is relevant for the development of WHO guidelines at the global level, as well as their adaptation at the national or subnational levels, where a wide array of stakeholders with diverse sets of values should be involved.77 78 In our overview of systematic reviews, we distilled procedural criteria (Stratil et al, forthcoming) and suggest that these be reviewed separately to inform guideline development and other health decision processes.18 41 42 45 75 76 79 We also refer to evidence-informed deliberative processes, which explicitly integrate the use of substantive criteria with procedural criteria to set priorities at national and subnational levels.80–82
The WHO-INTEGRATE framework is a highly interdisciplinary framework: each criterion, especially those criteria that are less developed in current EtD frameworks (eg, human rights and sociocultural acceptability) or absent from the literature (eg, societal implications), merits research to unpack them and, where applicable, provide a more detailed normative justification. We anticipate constructive input from and exchange with relevant disciplines, in particular public health ethics but also sociology, environmental sciences, economics and many others. Future collaborative research is expected to lead to a WHO-INTEGRATE framework version 2.0. This may advance the criteria and subcriteria and their normative foundations, as well as methodological approaches to populate these criteria with evidence.
To examine the value of the WHO-INTEGRATE framework to potential users, we conducted empirical qualitative research. Insights from interviews with key informants in relation to their recent experiences with developing WHO guidelines led to several refinements in the wording of the criteria and subcriteria and highlighted the importance of providing example questions as well as suggested methods. We expect that the second empirical qualitative research component, focus group discussions in Nepal, Uganda, Germany and Brazil, will yield additional insights from different perspectives and possibly further modifications to the framework. An integrated analysis of the views of WHO guideline developers and users will be published separately (Stratil et al, forthcoming).
Several of our key informants expressed concern about the potential workload resulting from collecting evidence for each of the criteria and, in particular, for the many subcriteria in the WHO-INTEGRATE framework. Both the process of prioritisation and the process of collecting evidence—through high-quality evidence synthesis or more pragmatic approaches—need to be tested in practice. We anticipate sharing worked examples and developing additional guidance on how to implement the framework in practice.
Conclusions
The WHO-INTEGRATE framework represents a comprehensive EtD framework rooted in WHO norms and values that is, in principle, suitable for individual-level, population-level and system-level health interventions that may or may not be characterised by complexity. It offers structured definitions for each of the six substantive criteria as well as the meta-criterion quality of evidence; example questions and suggested methods are provided to facilitate uptake. Importantly, this framework is intended to be used from the beginning and throughout a guideline or other health decision-making process, whether this process takes place at the global, national or subnational level. In working towards version 2.0, we welcome learning from the experiences of those applying the framework, as well as from researchers in disciplines concerned with the included criteria or subcriteria.
Acknowledgments
In our search for a suitable name that would transport the ideas contained within the new EtD framework, we adopted a name previously coined for the EU-funded INTEGRATE-HTA project, which two authors on this paper, EAR and RB, were part of and influenced by. We would like to thank INTEGRATE-HTA colleagues who were happy for us to use the project name in a different but related context. We are grateful to all participants in the key informant interviews. We are indebted to the many individuals who gave freely of their time and who thought ’out of the box' to raise important questions about and provide constructive criticism on previous versions of this paper: Saskia den Boon, Randy Elder, Nathan Ford, Elizabeth Fox, Davina Ghersi, Claire Glenton, David Gough, Sean Grant, Thomas Harder, Mike Kelly, Etienne Langlois, Fergus Macbeth, Paul Montgomery, Ani Movsisyan, Pierre Ongolo-Zogo, Deepak Paudel, Jennifer Petcovic, Mark Petticrew, Kevin Pottie, Gerard Schmets, Elham Shakibazadeh, James Thomas, Rebekah Thomas-Bosco, Andrea Tricco, Vivian Welch, Verina Wild and Teun Zuiderent-Jerak. We would also like to thank those colleagues who specifically provided input towards the table on how decision criteria may be populated with evidence: Kristin Bakke-Lysdahl, Björn Hofmann, Stephanie Levy, Graham Moore, Jane Noyes, Wija Oortwijn, Pietro Refolo and Dario Sacchini.
References
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.↵
- 6.↵
- 7.↵
- 8.↵
- 9.↵
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.↵
- 24.↵
- 25.↵
- 26.↵
- 27.↵
- 28.↵
- 29.↵
- 30.↵
- 31.↵
- 32.↵
- 33.↵
- 34.↵
- 35.↵
- 36.↵
- 37.↵
- 38.↵
- 39.↵
- 40.↵
- 41.↵
- 42.↵
- 43.↵
- 44.↵
- 45.↵
- 46.↵
- 47.↵
- 48.↵
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.↵
- 54.↵
- 55.↵
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.↵
- 67.↵
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.↵
- 78.↵
- 79.↵
- 80.↵
- 81.↵
- 82.↵
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93.
- 94.
- 95.
- 96.
- 97.
- 98.
- 99.
- 100.
- 101.
- 102.
- 103.
- 104.
- 105.
- 106.
- 107.
- 108.
- 109.
- 110.
- 111.
- 112.
- 113.
Footnotes
EAR and JMS contributed equally.
Handling editor Seye Abimbola
Contributors EAR conceived the overall research project with substantial input from JMS. RB, IBS, AP and SLN all provided important contributions to the overall direction of the project and to the draft manuscript. JMS undertook most of the work in step 1, with EAR and RB duplicating relevant elements in the process and all three analysing the findings. EAR implemented step 2a. JMS designed step 2b with input from EAR, collected the data and jointly analysed them with IBS. EAR, with input from AP, undertook the work in step 3. All authors discussed different versions of the framework, making revisions to criteria, definitions, subcriteria and example questions in an iterative manner. EAR wrote the manuscript, with JMS drafting selected sections; all authors critically reviewed the different versions of the manuscript, suggested revisions and approved the version to be published.
Funding Funding provided by the World Health Organization Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation. The development of this framework would not have been possible without the financial and, even more so, intellectual and procedural, support from the WHO, in particular from AP and SLN. We also gratefully acknowledge that JMS’s position was funded by the Bavarian Health and Food Safety Authority, and that IBS’s input was funded by the Norwegian Agency for Development Cooperation (NORAD).
Disclaimer The USAID, the Bavarian Health and Food Safety Authority, and NORAD had no influence on the research process or content of this manuscript. SLN and AP are staff members of the WHO. The authors alone are responsible for the views expressed in this publication, which do not necessarily represent the decisions or policies of the WHO.
Competing interests SLN helps oversee the quality of WHO guidelines and reports being a member of the RIGHT Working Group and the GRADE Working Group. EAR is a member of the GRADE Working Group.
Patient consent Not required.
Ethics approval Ethical approval was obtained from both the WHO Ethics Review Committee and the Ethics Committee of the Ludwig-Maximilians-Universität (LMU Munich), Germany.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.