Practice•

A framework for identifying and learning from countries that demonstrated exemplary performance in improving health outcomes and systems

Abstract

This paper introduces a framework for conducting and disseminating mixed methods research on positive outlier countries that successfully improved their health outcomes and systems. We provide guidance on identifying exemplar countries, assembling multidisciplinary teams, collecting and synthesising pre-existing evidence, undertaking qualitative and quantitative analyses, and preparing dissemination products for various target audiences. Through a range of ongoing research studies, we illustrate application of each step of the framework while highlighting key considerations and lessons learnt. We hope uptake of this comprehensive framework by diverse stakeholders will increase the availability and utilisation of rigorous and comparable insights from global health success stories.

Summary box

  • Mixed methods research provides a powerful tool for studying research questions, and stories of success among ‘exemplars’ provide invaluable learnings for replicating and accelerating health or health systems gains across contexts.

  • The Exemplars in Global Health (EGH) Partnership presents a systematic and comprehensive framework for conducting mixed methods research (including methods for identifying exemplar countries, selecting and engaging topic experts, collating and summarising evidence, undertaking qualitative and quantitative analysis, and preparing dissemination products for diverse audiences) when studying health and health systems outcomes in low and middle-income exemplars.

  • The EGH mixed methods approach to case studies can be used by diverse groups (including academics, NGOs and policy makers) to study and understand improvements in the most pressing health outcomes and implementation strategies in low and middle-income countries.

  • Learning from these stories of success could enable prioritisation of investments and programmatic action towards achieving the Sustainable Development Goals 2030.

Introduction

Starting in the early 2000s, the Millennium Development Goals (MDGs) aligned countries towards achieving measurable targets for health and survival (box 1). By the end of the MDG period, several health success stories had emerged. These exemplars provided an opportunity to understand ‘what works’ for attaining the Sustainable Development Goals (SDGs) (box 1).1–3

Box 1

The Millennium Development Goals (MDGs) and Sustainable Development Goals (SDGs)

In the early 2000s, the world adopted eight MDGs to achieve by 2015 and aligned on 21 targets to track progress.39 Several of these targets are relevant to health, either directly or indirectly. Three of these health-relevant targets – reduced incidence of malaria, reduced incidence of tuberculosis, and improved access to safe drinking water – were achieved at a global level; yet, several targets were missed, including reduction in child mortality (Goal 4), improvements to maternal health (Goal 5), reduction in the spread of HIV/AIDS (Goal 6) and sustainable access to sanitation (Goal 7). Targets to halve the proportion of individuals suffering from hunger were also not met. However, several regions and countries have achieved individual success in one or many of these indicators.

To build on and scale up momentum from the MDGs, in 2015, the United Nations General Assembly set 17 Sustainable Development Goals (SDGs) to achieve by 2030. These goals resulted from consensus derived from extensive multisector and multi-stakeholder consultations. Several of the SDGs are either directly or indirectly related to human health, including the following:

  • SDG 2. Zero hunger: End hunger, achieve food security and improved nutrition and promote sustainable agriculture (indicators include stunting, wasting, and overweight)

  • SDG 3. Good health and well-being: Ensure healthy lives and promote well-being for all at all ages (indicators include maternal mortality ratio, under-five mortality rate, vaccine coverage, and health worker density)

  • SDG 6. Clean water and sanitation: Ensure availability and sustainable management of water and sanitation for all (indicators include proportion of population using safely managed drinking water and sanitation services)

Many groups, such as the Countdown to 2030 collaborative and WHO success stories initiative, have highlighted policy successes to facilitate adoption of lessons in other contexts.4–12 Building on such efforts using standardised, rigorous and holistic mixed methods approaches can further the evidence base and ensure comparability across contexts. The Exemplars in Global Health (EGH) Partnership (established in 2017) aims to do precisely this (box 2). EGH conducts systematic and comprehensive mixed methods studies in priority health and development areas to generate evidence of impact at scale. We hope that lessons from ‘exemplar’ countries, which have achieved success in given areas (eg, child survival and nutrition), will support evidence-based decision-making among donors, governments and the global community.

Box 2

What is Exemplars in Global Health?

Exemplars in Global Health (EGH) is a partnership between academic researchers, implementers, policymakers, and donors that seeks to capture lessons from positive outlier countries that have achieved success in either solving health challenges in low-resource contexts or implementing programs at scale. To date, several EGH projects are ongoing based on an initial selection of diverse priorities spanning health, nutrition and key interventions within health systems: Stunting, Under-5 Mortality, Community Health Workers, Mass Drug Administration, and Vaccine Delivery.

Each area has seen tremendous improvement in the past two decades. A core set of principles drives EGH work and gave rise to the framework that we draw upon across our research. We aim to understand national and subnational successes and positive deviant examples through analysis that is: (a) methodologically rigorous; (b) objective; (c) comparable across countries; and (d) conducted in close partnership with in-country experts. EGH case studies attempt to capture not only the “what,” but also the “how” behind successful strategies and interventions (e.g., decision making, strategies and tactics, adaptation of programs over time). In sharing these lessons, we aim to enable policy makers, funders, global stakeholders, and implementing organizations to identify relevant lessons learned, and to adapt and emulate successes.

The EGH approach is rooted in the recognition that data and research from isolated methods are useful, yet incomplete in revealing the full picture. Hence, EGH applies comprehensive mixed methodologies while ensuring triangulation and corroboration of inferences across research activities and with input from technical and national experts. We believe this holistic process reveals novel and nuanced insights, while humbly acknowledging methodological and data limitations.

This paper presents our mixed methods research approach for studying positive outlier countries (figure 1). We discuss the process of identifying exemplar countries, selecting and engaging topic experts, collating and summarising available evidence, undertaking qualitative and quantitative analyses, and preparing dissemination products for diverse audiences. Of note, ours is far from the first framework developed to collect and analyse health data for use by policy makers. In fact, there is a rich tradition of similar methods used across social science disciplines.13–17 We have, however, formalised a mixed methods approach and applied it consistently across case studies of different topics and countries. To demonstrate, we share illustrative examples from ongoing EGH research. In describing the EGH framework, we encourage its uptake by diverse stakeholders when narrating the stories of positive outliers.

Framework components

EGH employs a framework to identify exemplar countries for a given topic, and then to assemble, generate, analyse and synthesise evidence for disseminating their stories (figure 1). Before identifying exemplar countries, a global health topic area —either a health outcome or an evidence-based health programme—is selected via consultations between subject matter experts and funding partners while considering relevance and evidence of impact in low-resource settings. The topics selected to date are detailed in box 3. Below, we provide a step-by-step overview of how we execute a common EGH research approach. Methods details for the various EGH topics are included in online supplemental file 1.

Box 3

Exemplar Types

Exemplar topics fall into two broad categories: outcomes and programs. Outcome topics investigate countries’ efforts in addressing a specific health outcome (e.g., stunting, under-5 mortality). Program topics look at which countries were most successful in implementing specific evidence-based programs, interventions, policies or practices (e.g., vaccine delivery, community health worker programs). For program topics, we focus less on causal evaluation of program impact and more on lessons in implementation.

Ongoing EGH Projects by Type

Health Outcomes:

  • Stunting

  • Under-5 mortality

  • Neonatal and maternal mortality

Programs:

  • Community health workers

  • Mass drug administration

  • Vaccine delivery

Figure 1
Request permissionrequest permission icon
Figure 1

Exemplars in Global Health research approach.

Technical Advisory Group selection and activities

For each topic, we convene Technical Advisory Groups (TAGs) of global experts to provide support throughout the research process. The role of a TAG is to ensure rigour, offer new ideas and avenues for exploration, and facilitate connections with in-country research partners and dissemination partners. In practice, similar to other groups that have used TAGs,18 EGH selects advisers who are renowned content and methods experts and represent a diverse group of stakeholders. TAG members are identified by project teams through self-nomination or through snowball sampling approaches, whereby experts are recommended by others in the field. The TAG is continuously engaged through all aspects of the project, including methods and inferences, strengths/limitations, prior knowledge, discourse on dissemination and partnerships.

Exemplar country selection

EGH selects exemplar countries based on their exceptional performance relative to secular or global trends. Historically, economic development has been a primary driver of improvements in health, but a close look at the data reveals that health outcomes vary considerably between countries with similar economic conditions and trajectories. The EGH approach aims to identify true positive deviants by selecting countries that outperform expectations from economic gains alone. Therefore, we select exemplar countries by assessing performance conditional on changes in the most relevant economic measures: gross domestic product (GDP) or gross national income per capita, programme-specific expenditures or other proxies for a country’s capacity to achieve improvement. The selected topics are studied across the past two decades, when data is of reliable quality and when significant improvements in the outcome are observed.

For some topics, we use a range of country-level time series estimates from multiple institutions. As shown in figure 2, an initial critical step is to generate visuals (eg, scatter plots) of average annual rate of change (AARC) in the outcome of interest as a function of the AARC in an economic indicator (eg, GDP per capita). This comparison enables identification of countries’ actual change relative to their expected change in the outcome given economic improvement only. Inspection of these quantitative measures is combined with qualitative considerations regarding feasibility of conducting research in-country, the generalisability of findings from a country given its size, diversity of government type, policy relevance and, when appropriate, representation across geographic regions. The shortlist of countries is shared with the TAG, which subsequently ranks contenders for the final list of exemplar countries. In the case of our under-five mortality research, this process resulted in the selection of seven exemplar countries, shown in figure 2.

Figure 2
Request permissionrequest permission icon
Figure 2

Exemplar country selection process: under-five mortality example. GDP, gross domestic product.

Evidence collection and collation

Understanding the complex chain of context, policies, programmes and interventions that drive large-scale national change is challenging. And although establishing causality with retrospective studies is aspirational, triangulating evidence across a broad range of data sources and methods brings us closer to the truth (see figure 3 for an example of under-five mortality data triangulation in Rwanda). The EGH approach to evidence collection is to first systematically and comprehensively review existing evidence to identify knowledge gaps that could be filled with empirical analysis. Subsequently, we triangulate analyses from qualitative, quantitative and policy data to uncover a holistic narrative of what contributed to a given country’s success. In addition to critical insights from the TAG, an appropriate in-country partner is identified for each exemplar country; this partner plays a key role in collecting, generating and synthesising evidence across analyses. To illustrate, each of the evidence collection steps is discussed through EGH research examples below.

Figure 3
Request permissionrequest permission icon
Figure 3

Triangulation across data: under-five mortality example, Rwanda.

Literature review

The EGH approach involves conducting one or more literature reviews for the topic of interest. The type of literature review conducted is dependent on several factors, including (1) the existing breadth of information available; (2) the need to assess evidence systematically versus more broadly; (3) the intended final output (eg, a meta-analysis or descriptive summary); and (4) the available time and financial resources. The main objective of the literature review is to assemble the latest evidence on the topic of interest that can be cited and from which insights can be extracted and gaps identified. Detail on the literature review features of EGH projects is provided in box 4.

Box 4

Literature review features of EGH projects

The EGH approach encourages use of one or multiple literature review methodologies as appropriate to the question at hand. It is typical to deliberate such decisions a priori and adjust the literature review scope to research needs. For instance, in some cases, a systematic literature review may be the best approach. If the topic scope is too large, however, targeted searches may be more practical and allow for deeper dives into areas that emerge as important after an initial, more cursory literature review.

The EGH Stunting Project, for example, aims to collate causal evidence on “factors that were linked to stunting decline.” Given the existing breadth of literature on this topic in many countries, in addition to the need to assess quantitative associations systematically and objectively, researchers opt to conduct a thorough systematic literature review inclusive of statistical meta-analysis when possible. They explore more then 15 online databases (e.g., PubMed, Scopus, CINAHL) for peer-reviewed literature and more than 10 grey literature repositories (including government websites, NGO databases, and UN websites). They set inclusion and exclusion criteria (against which articles were identified, screened, and shortlisted for abstraction) and also conduct hand-searches of bibliographies. This process, applied to literature assessing stunting reduction determinants in Peru from 1990-2018, yielded 500 unique records, of which 159 underwent full text review, and of which 141 relevant studies were ultimately included in the final literature synthesis.

The EGH Under-5 Mortality Project, in contrast, opted to conduct a broad, iterative scoping review of literature using similar databases. This rapid review enabled researchers to identify emergent themes within a vast existing evidence base, while concurrently planning research goals and methods to fill existing evidence gaps. Similar work was done within the implementation science literature to identify relevant frameworks for adaptation.

The literature review can also reveal existing frameworks, models and methods for conducting research or analyses, to be considered for the topic of interest. For instance, the International Initiative for Impact Evaluation (3ie) produces evidence gap maps (EGM) that summarize evidence on the effects of development policies and programs in a particular sector, sub-sector, or thematic area, structured around a framework of interventions and outcomes.40 Previously developed EGMs, including those developed using the 3ie approach, on topics of interest may be a useful addition to the EGH toolkit. In the same vein, systematic tools for economic evaluation or other niche methods can be identified through this process.

As a final note, methods for synthesizing collated evidence dictated both by research objective and by available literature. The Vaccine Delivery Project team uses a literature review to assemble and harmonize operational frameworks for vaccine systems to develop a working model for the research. As shown in the EGH Stunting and Under-5 Mortality examples, the availability of several data-rich objective studies can facilitate quantitative meta-analyses, while qualitative reports may require syntheses for emergent themes. While a complete review of types, conduct, and syntheses of literature reviews is out of the scope of this protocol paper, we stress the importance of mobilizing literature review methodologies effectively at the outset and throughout the conduct of the case study to inform study planning and enrich results interpretation.

Qualitative analysis

EGH research involves conducting consultations with key informants. These consultations can be in-depth interviews and/or focus group discussions as required for the study question and target population. Potential informants include international experts, donors, researchers, representatives of multilateral or bilateral institutions, current and former Ministry of Health members, programme implementers such as national and international non-governmental organisations, front-line workers, and community members or direct beneficiaries of programmes implemented at scale at the local level. Snowball sampling was used across projects to identify additional key informants. Once data were retrieved, qualitative data analysis of the appropriate level of rigour was carried out according to an evidence-based conceptual framework.

The Community Health Worker (CHW) Project, which incorporates research visits to each exemplar country, illustrates how the EGH project collaborates with local partners. Prior to a research visit, the research team works closely with the local partner to identify stakeholders. These trips include site visits to observe programmes and in-person, semi-structured interviews with key stakeholders.

Quantitative analysis

Data sources

Although data quality is a substantial concern in many low-income countries, rich data sources for measures of health outcomes, health systems and contextual factors, including education, income distribution and population density, are plentiful. Household surveys are readily accessible and provide representative snapshots of changes in health and intervention coverage over time. Along with primary data sources, there are numerous sources that provide estimates of disease burden and intervention coverage which incorporate data into models that produce full time-series for every country. In addition, estimates summarising the state of a country’s economy, education system and infrastructure offer helpful context for describing the combination of factors that contributed to changes in outcomes. Critically, care must be taken when incorporating modelled estimates as inputs into analysis since these are developed based on already-known relationships.

Descriptive statistics

The aforementioned data sets enable independent calculation of relevant statistics and the development of insightful novel measures. Our work uses a number of novel measures produced internally, like the slope index of inequality and concentration index,19 to measure absolute and relative socioeconomic inequalities in health outcomes, as well as externally produced numbers, like geospatial estimates from the Institute for Health Metrics and Evaluation (IHME), which leverage location data to estimate the subnational distribution of outcomes at a 5×5 km level of granularity.20 Through the development of descriptive statistics that span multiple strata, we explore variation across a range of dimensions, including space, time, gender, economic status and more.

Causal quantitative evaluation

Causal evaluation combines data used in generating descriptive statistics with a hypothesised conceptual framework, enabling synthesis of multiple sources of quantitative data. There are numerous approaches to decomposing past outcomes into explainable and unexplainable factors, but EGH projects primarily use two types of approaches. One quasi-experimental approach involves using data from within the country being analysed and employing regression techniques in combination with hierarchical causal diagrams to identify effect sizes of each hypothesised causal factor. Alternatively, meta-analysis of pooled data from randomised control trials can produce estimates of risk factor effect sizes that are either the same across locations, or at least influenced by the observed effects in other countries.

In each approach, effect sizes are combined with estimates of changes in each explanatory factor to calculate how much each factor contributed to observed changes in the outcome of interest over the time period. The final product is a proportioning of explainable changes to a group of causal factors, many of which fall under the health system. Causal inference, already a difficult task, is further complicated by limited data and uncertainty about the pathways through which multiple factors combine to impact outcomes. Despite these concerns, causal evaluation can provide critical support for hypothesised high-impact activities.

Policy/programme analysis

Organising information on a country’s policies, programmes, and financing during the time period of evaluation can enable a clearer understanding of how that country was able to realise success. While details of some policies may be readily accessible without the support of in-country informants, conversations with key stakeholders often identify additional documentation of processes, financing and policies that were critical to the country’s success. Following up on these documents, combining them with the literature review and distilling them into a clear timeline of actions that each country took provides a clearer view of the policy mechanisms behind a country’s success. Extracting data on the financials behind policies can provide substantial additional insight, but often this is difficult to do given a lack of data availability. Drafting a policy timeline for expert stakeholders to review can be useful, as it enables them to point out missing initiatives that require further investigation. Multiple iterations of timeline review enable convergence towards a consensus timeline endorsed by experts (see figure 4 for a consensus stunting-related policy and programme timeline from Peru).

Figure 4
Request permissionrequest permission icon
Figure 4

Policy and programme timeline: stunting example, Peru.

Evidence synthesis and triangulation of evidence

While application of each of the aforementioned methods can independently contribute to the literature for a given country or topic area, our goal is to combine these pieces to tell comprehensive stories and fortify the weaknesses of each method. We undertake a rigorous and iterative triangulation process whereby each research output is synthesised collectively to build a coherent story of success factors. Although led by the research team, it is critical that input from country experts and TAG be incorporated during this process. To assist with the evidence synthesis process, it can be helpful to draw on pre-existing frameworks. For example, the CHW Project uses the ExpandNet scaling up framework to organise the outputs from each of the employed methods.21

Dissemination products

Finally, research on exemplars should be accessible by and applicable to a wide audience, for instance, through national level dissemination events geared towards policy and actionable decision-making to drive progress. Accordingly, the narrative can be assembled into a variety of dissemination products (see table 1 for a table of EGH dissemination products) that align with preferences of various target audiences (see figure 5 for an example of user testing conducted to identify preferred dissemination formats).

Table 1
|
Exemplars in Global Health output formats and intended audiences
Figure 5
Request permissionrequest permission icon
Figure 5

Insights on preferred types of content from user testing.

Several other outputs can be created to reach diverse audiences. Users of the EGH framework are encouraged to consider which products are best for their intended audiences. For instance, material from the CHW Project has been incorporated into ‘Strengthening Community Health Worker Programmes’, an open, online, university-level course hosted on HarvardX that reaches over 10 000 current and future health systems leaders in over 170 countries.

Finally, in-person and virtual discussions can serve as both opportunities for dissemination and means of refining research findings for policy and action. Significantly, it will be important for stakeholders to interpret and adapt information from exemplar countries to their respective contexts. EGH provides a range of interactive services to support this interpretation and adaptation.

Patient and public involvement

Our research question was developed to identify drivers of exemplary national performance. For this reason, patients were not involved in the design of the study.

Conclusion

Summary

The EGH framework offers guidelines and recommendations for conducting rigorous and objective research on global health success stories. With oversight by leading global experts, we propose mixed methods studies to develop systematic and comprehensive narratives of exemplar countries’ success stories. Data are presented in a way that enables other national or subnational governments, as well as global funding and implementing mechanisms, to prioritise strategies proven to work at scale. In short, EGH’s research-based implementation narratives explain how countries implemented, adapted and scaled interventions that work.

The merits of conducting systematic case studies to highlight global health successes is evident in the sheer number of previous published efforts. The Countdown to 2030 consortium22 has published dozens of country case studies, several as scientific articles.3 23–25 The International Food Policy Research Institute26 has published success narratives as book chapters, reports, brochures, and journal articles.27 Similarly, the 3-year multidisciplinary, multicountry series of ‘success factor’ studies coordinated by Partnership for Maternal, Newborn and Child Health (PMNCH), WHO, World Bank and the Alliance for Health Policy and Systems Research aimed to understand how some countries accelerated progress to reduce preventable maternal and child deaths.28–30 The EGH framework builds on and consolidates lessons from these efforts, positing one common systematic approach that can be used across diverse topics and research groups.31–35

Each of the framework elements aims to complement one another and strengthen emergent themes. We assemble topical technical advisors at the outset of each project. Their role is critical in identifying exemplar countries, fostering in-country partnerships, overseeing methods design and analyses, and commenting on interpretation of final results. Conducting an environmental scan of existing evidence shapes the required methodological approaches, provides context for research teams and technical advisors, and informs the overall research process and final results synthesis. Undertaking robust quantitative, qualitative and policy analyses provides the most useful inputs to understanding the exemplar story. Triangulation of all existing and novel analyses under the oversight of technical advisors and country experts is the critical final step in developing evidence-based narratives of success and actionable insights.

Implications of EGH approach

The merits of studying stories of success to understand transferable lessons cannot be understated. Methodologically rigorous, comprehensive and systematic narratives provide valuable insights that enable learnings from the past to be applied to current and future global health challenges. We believe that the EGH approach can be applied across diverse disciplines to catalyse such efforts. Importantly, the standardised EGH framework proposes cross-cutting principles while allowing room for flexibility by topic of research and/or researcher area of expertise.

That said, the EGH approach is merely a starting point. Additional steps are required to translate lessons into implemented policy. The EGH Partnership offers support for implementation in the form of direct technical assistance in policy-making and connection to experts. Other organisations provide similar support, as well as funding.

Limitations

Despite the potential of the EGH approach, several limitations are worth noting. First, retrospective analyses are challenging even in the most data rich settings. Often, direct measures of key variables are not collected, poorly measured or simply inaccessible due to poor institutional memory (eg, due to staff turnover) or data storage. Measuring some important constructs such as food security or quality of care is elusive and quantitatively challenging.36 Thus, reliance on ‘proxy’ measures is commonplace in retrospective studies, even though they may threaten reliability and accuracy if not carefully selected. Confounders of key associations may also be unavailable for analyses; therefore, exposure-outcome associations should be carefully interpreted. Additionally, quantitative analyses of ecological associations may be subject to ecological fallacy and the choice of statistical models may also influence results if incorrectly specified;37 conducting sensitivity analyses of key associations is thus critical to ensuring objective inferences. Key informants for qualitative research can be difficult to identify and connect with for interviews. Other important data (eg, dietary intake, from food frequency questionnaires) is almost always based on participant recall across long or short-term windows. Thus, recall bias in administered surveys and among interviewed country experts and beneficiaries is an ongoing challenge, and efforts to mitigate this bias in primary data collection must be considered.38 Triangulating qualitative and quantitative evidence mitigates risk of bias or uncertainty in inferences.

Future work

The EGH Consortium will continue to evolve and expand our research portfolio, with the framework presented in this paper underpinning all of our studies. Our aim is to not only provide funding agencies, multilateral agencies, governments and researchers with our learnings organised into a methodical framework, but also to share transferable and actionable knowledge for countries and foster partnerships with those interested in studying EGH. We also hope to promote improvements in the quality and comprehensiveness of data collection, as well as alignment on the appropriate quantitative methods for making inferences from data, by continuing to document EGH experiences in subsequent studies.

We believe that the methodical EGH approach to understanding case studies of successes in global health will provide objective evidence that can be used across diverse groups to accelerate improvements in the most pressing health outcomes and implementation strategies in low and middle-income countries. Learning from these stories of success could enable prioritisation of investments and programmatic action towards achieving the SDGs by 2030.

Article metrics
Altmetricopen-url
Dimensionsopen-url