Table 3

WHO-INTEGRATE framework version 1.0: criteria and suggested types of primary studies, evidence synthesis methods and approaches to assessing quality of evidence

CriteriaTypes of primary studies*Evidence synthesis or mapping methodsPragmatic approachesApproaches to assessing quality of evidence
Balance of health benefits and harms.
  • Efficacy or effectiveness on health of individuals/populations: RCTs, pragmatic trials, quasi-experimental studies, comparative observational studies; longer term observational studies, modelling (eg, transmission modelling for infectious diseases).

  • Patients’/beneficiaries’ values in relation to health outcomes: qualitative studies (eg, semistructured interviews, focus groups), cross-sectional studies.

  • Safety risk profile of intervention: RCTs, quasi-experimental studies, comparative observational studies for anticipated harms; registry studies, longer term observational studies, case series, case reports for unanticipated effects.

  • Broader positive or negative health-related impacts: RCTs, quasi-experimental studies, observational studies, qualitative studies.

  • Systematic reviews of efficacy/effectiveness83 for anticipated effects.

  • Qualitative evidence syntheses84 85 and mixed-method reviews86 or cross-sectional studies66 for patients’/beneficiaries’ values in relation to health outcomes.

  • Scoping reviews87 88 for unanticipated effects.

  • Rapid reviews of efficacy/effectiveness.89–91

  • Overviews of systematic reviews.83 92

  • GRADE.70 71 73

Human rights and sociocultural acceptability.
  • Accordance with universal human rights standards: mapping of relevant aspects, pro et contra analysis,93 ethical analysis (eg, casuistry, coherence analysis, wide reflective equilibrium),94 power analyses, human rights impact assessment.95

  • Sociocultural acceptability of intervention, impact on autonomy of concerned stakeholders, intrusiveness of intervention: mapping of relevant aspects, pro et contra analysis,93 discourse analysis, qualitative studies (ideally longitudinal to examine changes over time), discrete choice experiments, cross-sectional studies,66 longitudinal quantitative studies (to examine changes over time), mixed-method studies.

  • Ethics syntheses96 97 for accordance with universal human rights standards.

  • Qualitative evidence syntheses84 85 98 and mixed-method reviews86 for sociocultural acceptability and impact on autonomy of concerned stakeholders and intrusiveness of interventions.

  • Purposively selected studies from different contexts (to illustrate broad spectrum of issues).

  • GRADE CERQual72 99 (where applicable).

  • Q-SEA for ethics analyses.57

Societal implications.
  • Social impacts: RCTs, quasi-experimental studies, comparative observational studies, longitudinal implementation studies, qualitative studies, case studies, power analyses.

  • Environmental impacts: RCTs, quasi-experimental studies, comparative observational studies, longitudinal implementation studies, qualitative studies, case studies, environmental impact assessments, modelling studies.

  • Combined social, environmental and economic impacts: health impact assessments, modelling studies (eg, decision-analytical modelling).

  • Systematic reviews of effectiveness.83

  • Qualitative evidence syntheses.11 84 85

  • Mixed-method reviews.86

  • Health technology assessments.68

  • Purposively selected studies from different contexts (to illustrate broad spectrum of issues).

  • No standardised approach.

  • GRADE70 71 (where applicable).

Health equity, equality and non-discrimination.
  • Impact on health equality and/or health equity, distribution of benefits and harms of intervention: human rights impact assessment,95 disaggregated RCTs, quasi-experimental or comparative observational studies, RCTs and quasi-experimental or comparative observational studies conducted in disadvantaged groups,100 power analyses, GIS-based studies, qualitative studies, ethical analysis.

  • Affordability of intervention: cross-sectional or longitudinal observational studies, discrete choice experiments, qualitative studies, catastrophic health expenditure studies.

  • Accessibility of intervention: health system barrier studies, cross-sectional or longitudinal observational studies, discrete choice experiments, qualitative studies, ethical analysis, GIS-based studies.

  • Severity and/or rarity of the condition: health state valuations, cross-sectional studies for severity of condition; observational studies for frequency (incidence, prevalence) of condition.

  • Lack of a suitable alternative: situation analysis of intervention options; quantitative or qualitative studies of adverse effects of existing options.

  • Quantitative systematic reviews83 using PROGRESS101 or PROGRESS PLUS,102 where possible using prespecified subgroup analyses.

  • Quantitative systematic reviews targeting disadvantaged groups.

  • Equity weights and social welfare functions in economic analyses (see Financial and economic considerations).

  • Qualitative evidence syntheses11 84 85 and mixed-method reviews.86

  • Ethics syntheses.96 97

  • Purposively selected studies from different contexts (to illustrate broad spectrum of issues).

  • Scoping reviews.87 88

  • Overviews of systematic reviews.83 92

  • No standardised approach.

  • GRADE70 71 for subgroup analyses (where applicable).

  • Relevant considerations, such as including health equity as an outcome, in Welch et al.103

Financial and economic considerations.
  • Financial impact: prices and price justifications for unit costs per beneficiary/population according to relevant perspectives, budget impact analysis.104

  • Impact on economy: economic burden of disease studies,105 quasi-experimental studies, comparative observational studies, longitudinal implementation studies, qualitative studies, case studies, modelling studies.

  • Ratio of costs and benefits: economic analyses as a comparative analysis of alternative courses of action in terms of their costs and consequences (eg, cost-minimisation analysis, cost-effectiveness analysis, cost-utility analysis, cost-benefit analysis).

  • Comprehensive or representative cost or budget impact data at the appropriate level (global, regional, national, subnational).

  • Economic burden of disease studies undertaken at the appropriate level (global, regional, national, subnational).

  • Economic analyses undertaken at the appropriate level106 107 or economic analysis reviews.108–111

  • Cost or budget impact data for purposively selected contexts.

  • Economic analyses undertaken for selected contexts.

  • No standardised approach.

  • Relevant considerations in Drummond et al106 (chapter 3 and box 3.1) and Brunetti et al.112

Feasibility and health system considerations.
  • Legislation, leadership and governance, interaction with and impact on health system, need for, usage of and impact on health workforce, human resources and infrastructure: health systems research,113 including mapping of relevant aspects, situation analysis, cross-sectional studies, qualitative studies, case studies.

  • Qualitative evidence syntheses,11 84 85 mixed-method reviews.86

  • Formal consultation of content experts.

  • No standardised approach.

  • GRADE CERQual72 (where applicable).

  • *This table offers a collection of suitable methods rather than guidance on the most appropriate method, which depends on the specific research question. Where appropriate, the order in which the methods are presented implies a hierarchy of evidence (eg, RCTs are more suited to assessing questions of efficacy than modelling).

  • GIS, geographical information system; GRADE, Grading of Recommendations Assessment, Development and Evaluation; GRADE CERQual, Confidence in the Evidence from Reviews of Qualitative Research; HTA, health technology assessment; INTEGRATE, INTEGRATe Evidence; Q-SEA, Quality Standards for Ethics Analyses in HTA; RCT, randomised controlled trial.