High-performing primary health care (PHC) is essential for achieving universal health coverage. However, in many countries, PHC is weak and unable to deliver on its potential. Improvement is often limited by a lack of actionable data to inform policies and set priorities. To address this gap, the Primary Health Care Performance Initiative (PHCPI) was formed to strengthen measurement of PHC in low-income and middle-income countries in order to accelerate improvement. PHCPI’s Vital Signs Profile was designed to provide a comprehensive snapshot of the performance of a country’s PHC system, yet quantitative information about PHC systems’ capacity to deliver high-quality, effective care was limited by the scarcity of existing data sources and metrics. To systematically measure the capacity of PHC systems, PHCPI developed the PHC Progression Model, a rubric-based mixed-methods assessment tool. The PHC Progression Model is completed through a participatory process by in-country teams and subsequently reviewed by PHCPI to validate results and ensure consistency across countries. In 2018, PHCPI partnered with five countries to pilot the tool and found that it was feasible to implement with fidelity, produced valid results, and was highly acceptable and useful to stakeholders. Pilot results showed that both the participatory assessment process and resulting findings yielded novel and actionable insights into PHC strengths and weaknesses. Based on these positive early results, PHCPI will support expansion of the PHC Progression Model to additional countries to systematically and comprehensively measure PHC system capacity in order to identify and prioritise targeted improvement efforts.
- Primary care
- primary health care
- global health
- universal health coverage
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
Effective primary health care (PHC) is essential for achieving the promise of quality universal health coverage, but PHC performance is weak in many low-income and middle-income countries (LMICs).
The ability to improve PHC systems is limited by a lack of relevant metrics and data for assessing critical areas of PHC performance and system capacity.
The Primary Health Care Performance Initiative (PHCPI) undertook a structured, participatory process to design a new mixed-methods assessment tool to measure PHC capacity in LMICs more systematically and comprehensively.
PHCPI partnered with governments in five LMICs to pilot the assessment tool in 2018 and found that the assessment yielded novel and actionable information on PHC strengths and weaknesses and was feasible, acceptable, and effective in generating local ownership of the results.
The 2018 Astana Declaration reaffirmed the global community’s commitment to strengthening primary health care (PHC) and highlighted the central role that PHC must play in the achievement of universal health coverage.1 However, too often PHC is weak, underprioritised, and unable to deliver on this potential, particularly in low-income and middle-income countries (LMICs).2–4
In many LMICs, the ability to improve PHC is limited by a lack of relevant data to accurately measure and diagnose performance to inform policies and set priorities. The Primary Health Care Performance Initiative (PHCPI) was formed to accelerate improvements in PHC through better measurement and knowledge sharing.5 To guide its work, PHCPI developed a conceptual framework outlining the core systems, inputs, and service delivery elements necessary to produce strong PHC outputs and outcomes (figure 1)6 and conducted an extensive scoping process to identify the best globally available metrics to assess each component captured in the conceptual framework. Through these efforts, PHCPI was able to identify adequate measures for many components of the framework, including topics such as PHC spending, access, quality, service coverage, and health outcomes.7
However, PHCPI’s scoping process failed to identify robust, globally available indicators of the underlying capacities of PHC that impact overall performance and outcomes (figure 1).7 As described further in box 1, PHC capacities include topics such as:
Key questions assessed by the primary health care (PHC) Progression Model
Governance and leadership
Do countries have evidence-based PHC policies and strategies in place?
Are there effective governance structures to implement and enforce these PHC policies?
Is there a robust quality management infrastructure for PHC, including quality policies and strategies, legislation and regulation, quality standards, and use of continuous quality improvement programmes and methods?
Does the country have a system that formalises and ensures strong social accountability mechanisms, including the systematic engagement of private sector, civil society, non-governmental organisations, and non-health actors in the integrated planning and governance of PHC and public disclosure of performance?
Adjustment to population health needs
Do countries have comprehensive and reliable surveillance systems in place to detect and respond to changing disease burden and emerging outbreaks?
Are national health priorities set based on disease burden, health outcomes, and user needs?
Does the PHC sector have a learning system that prioritises continual reflection and improvement?
Are key inputs—including drugs and supplies, facility infrastructure, information systems, health workforce, and funds at the facility level—available?
Are they equitably distributed?
Are they of sufficiently high quality to meet population health needs?
Population health management
Are local priorities evidence based and determined in collaboration with local communities and stakeholders?
Do communities have input to and impact on the way that PHC is financed, governed, and implemented?
Is a system of empanelment, or rostering, in place to ensure that the entire population is known to the health system and that specific service providers have responsibility for specific panels of patients?
Does proactive population outreach occur to deliver essential health services to those in need?
Facility organisation and management
Are primary care services organised and delivered by effective provider teams, capable of ensuring comprehensive and coordinated care?
Are facilities effectively led by managers with the ability to organise operations, motivate staff, and deploy resources?
Do facilities set performance targets, have staff capacity to capture and use data at the point of care to monitor and improve performance, and implement quality improvement activities?
Is supportive supervision routinely conducted?
How well the PHC system is governed and led.
Whether PHC systems have the ability to detect and adjust to changing population health needs.
Whether key inputs are available, equitably distributed, and of sufficiently high quality to meet population health needs.
Whether PHC systems know and engage with the populations they serve.
Whether healthcare providers work as teams in well-managed facilities, using data to drive improvement efforts.
Many of these are complex, interrelated topics whose performance varies over a continuum that is difficult to quantify and requires more nuanced exploration than a single quantitative measure may allow.
Through engagement and partnership with representatives from more than 30 LMICs over several years, PHCPI recognised that the collective inability to fully measure PHC capacity severely curtailed countries’ ability to develop a holistic understanding of their system’s strengths and weaknesses and hence their ability to identify and implement needed improvements. To address this measurement gap and ensure that essential information on PHC capacity could be comprehensively assessed, PHCPI developed a novel tool—the PHC Progression Model—to systematically assess the complex, foundational capacities of PHC. The results of the PHC Progression Model assessment are incorporated into the PHC Vital Signs Profile, a PHCPI measurement tool that summarises a country’s performance across the conceptual framework and is designed to support countries in identifying priority areas for improvement in PHC, track progress over time, and promote accountability for results by making essential performance information transparent and publicly available.8
In 2018, PHCPI partnered with five LMIC to pilot PHC Progression Model assessments. This paper describes the methods used to develop the PHC Progression Model as well as the process and lessons learnt from implementation.
A new way of measuring PHC capacity
To identify a means of systematically assessing PHC capacity, PHCPI built on the recent increase in use of mixed-methods, rubric-based assessment tools in the health landscape. These tools are designed to capture performance across a range of different levels of system maturity, as defined by prespecified performance categories described in a series of rubrics. As a first step in conceptualising the PHC Progression Model, we used literature reviews, internet searches, expert recommendations, and snowball sampling to search for rubric-based tools in the healthcare domain. We identified a multitude of relevant tools, and six in particular that substantially influenced the conceptualisation and design of our work.9–14 Through a review of publicly available materials and interviews with the developers and/or implementers of these tools, we extracted relevant key learnings and the best practices around tool design and implementation (box 2). Additional information about these tools is available in online supplementary file 1.
Relevant lessons learnt on mixed-methods, rubric-based tool design and implementation from a review of existing tools
Rubric and standards-based tools can be used to assess across multiple, related domains.
Most assessment tools employ a four or five-level categorisation, which enables nuanced descriptions of performance levels and/or progressive levels of maturity that can more completely capture complex topics and progression over time than simple binary indicators.
These tools can be completed based on user opinion, qualitative data, quantitative data, document review and/or combinations of any of the above. Assessment can be internally driven by the stakeholders who participate in the activity being assessed or conducted by an objective, external body. Decisions around which methodology(ies) to use should be driven by the objectives of the assessment—for example, for comparison across sites or internal quality improvement purposes.
The time and resources required to complete assessments are closely linked to the methods used.
The assessment tool can be a vehicle for bringing together diverse stakeholders whose work relates to the same topic or output, but who may not have the opportunity to regularly interact with and learn from one another.
Based on these lessons learnt, PHCPI decided that the new assessment tool should be implemented through a joint internal and external assessment process led by in-country teams who rigorously document findings to collectively determine scores that are subsequently reviewed by an external team to ensure that results are rooted in evidence and that performance standards are consistently applied across countries. Such a process offered the best means of achieving PHCPI’s two goals for the assessment: (1) to produce a national assessment of PHC capacity that is acceptable to and owned by policymakers in LMIC and used to drive improvement efforts and (2) to drive accountability and improvement through the public release of Vital Signs Profiles that are standardised across countries and enable peer-to-peer benchmarking and learning.
Development of the tool
PHCPI undertook a structured, iterative, and participatory process to develop the PHC Progression Model (figure 2). The first step was a targeted review of relevant global health frameworks, toolkits, and data collection instruments15–20 to identify any indicators or normative standards related to the content to be assessed via the PHC Progression Model. Drawing where possible from these validated tools and supplementing with qualitative indicators to create a set of rubrics outlining four progressive performance categories for each topic assessed, we completed a first draft of the PHC Progression Model. To assess the face validity of the tool, we then partnered with the Alliance for Health Policy and Systems Research at the World Health Organization (WHO) to complete mock assessments using the seven Primary Health Care Systems case studies that were available at the time (Bangladesh, Ethiopia, Ghana, Kenya, Nigeria, Tanzania and Uganda).21 Based on the findings of this exercise, we made adjustments and generated a second draft of the tool.
Next, we identified 75 individuals with expertise in specific areas covered by the PHC Progression Model and conducted key informant interviews and online surveys. Each expert was asked to review the sections of the tool relevant to their area of expertise and reflect on the following dimensions of each measure: relevance, calibration, reliability, and comprehensiveness.
Finally, we convened an expert advisory group made up of international measurement experts as well as policy-makers from seven LMICs to review the content of the PHC Progression Model and assess whether each measure included in the tool captured the right information, was clear and well calibrated, and was feasible to assess in the represented countries. Based on all expert input received, we updated the PHC Progression Model to a version ready for pilot testing.
The PHC Progression Model
The resulting PHC Progression Model is a mixed-methods assessment tool for measuring the foundational capacities of PHC.22 The model consists of 32 measures covering the content areas described in box 1. As shown in figure 3, each measure includes a rubric that is used to assign a country to one of four performance categories ranging from level 1 (low) to level 4 (high). The criteria for levels 1–4 vary according to the content of each measure being assessed; individual criteria can be found in the assessment tool.22 Data and evidence for completing the assessment can be drawn from a variety of sources available within countries, including policy documents; routine reports and assessments; data elements from global surveys and/or locally owned and generated data; and key informant interviews with a variety of public and private, governmental and non-governmental organisations including at the subnational levels, as appropriate depending on the measure to ensure inclusion of diverse perspectives.
Scoring of each measure of the PHC Progression Model employs a threshold approach, in which a performance level can only be achieved if all criteria described in the measure meet the performance described in the corresponding rubric. The 32 measure scores are summarised into nine subscores, corresponding to each subdomain of the PHCPI conceptual framework being assessed, by taking a simple unweighted average of all the constituent measures. The nine subscores are then summarised into three overall scores—Governance, Inputs, and Population Health and Facility Management—for display in the Capacity pillar of the Vital Signs Profile, again by taking simple unweighted average of the constituent subscores (figure 3).
Piloting of the PHC Progression Model
PHCPI partnered with national Ministries of Health in five LMICs—Argentina, Ghana, Rwanda, Senegal, and the United Republic of Tanzania—to pilot a PHC Progression Model assessment in 2018. Countries were selected based on a formal expression of interest by the minister on behalf of their government, strong pre-existing relationship between one of the PHPCI partners and the ministry, and availability of resources to support the assessment.
Each country formed a core team responsible for implementing the assessment in accordance with the methods and standards outlined in a standardised assessment guide. As described in table 1, the size and composition of the core team varied by country depending on factors such as the time and availability of technical staff and preferences for inclusivity across multiple organisations or divisions within the Ministry of Health. Most often, the core team contained technical staff from Ministries of Health with expertise in and oversight of PHC in their country, as well as technical consultants who were often responsible for data collection.
The assessment team began by contextualising the assessment to the country context, a process which entailed agreeing to local definitions for: (1) the package of services considered to make up ‘PHC’, (2) the facilities considered to be ‘PHC facilities’ and (3) the human resources for health considered to be ‘PHC human resources’. Next, country teams undertook a detailed review of the types of data that would be required to score each measure, identified potential quantitative and qualitative sources, and made a plan for efficiently collecting needed data. When key informant interviews at the subnational level were deemed necessary, country teams also developed a sampling strategy to ensure that subnational data sources would yield a representative picture of what was truly occurring in the country.
Countries approached data collection in diverse ways, with some choosing to centralise the process within one or two individuals and others distributing responsibility to multiple members of the assessment team. Teams based decisions around considerations of feasibility, acceptability and effectiveness, including: how to structure their assessment process based on experience with similar assessment methods, for example, the Joint External Evaluation12; the time and resources key stakeholders could commit to the process; and an understanding of what steps would be necessary to generate local ownership of the results and be most likely to encourage the use of results to drive improvement efforts. Table 1 summarises the different assessment processes undertaken by each country team.
Data collection plans were approved by stakeholders such as high-level representatives from Ministries of Health and core implementing partners whose acceptance of the results would be critical for ensuring resulting data would be used to drive improvement efforts. Teams then completed data collection and used templates provided by the PHCPI team to compile and synthesise relevant information from across all data sources for each measure.
These data syntheses were then used as the evidence base for an internal scoring exercise, which consisted of convening a stakeholder group to review all of the assembled evidence for each measure and use the rubric to assign the country’s performance to one of the four performance categories. In the event that assessment teams had been unable to identify sufficient data to score a measure, a score of level 1 was assigned. (In Argentina, the federal administrative nature of the health system and current events made assessment of Population Health and Facility Management unfeasible, and results were instead displayed as ‘N/A’.) Participants in the scoring exercise were selected by the assessment team based on how best to generate buy-in for and acceptance of the results.
Next, the results of the internal scoring exercise, along with all supporting evidence, were shared with the PHCPI country engagement lead and a team from PHCPI partner organisation Ariadne Labs for external validation. The goals of the external validation were to ensure that the available evidence justified the scores given by the country team and that measurement standards were being consistently applied across countries. Often, the external assessment process resulted in the identification of measures where more detailed evidence was needed to justify the internal scores; in these cases, the external and country assessment teams would engage in ongoing dialogue and review of additional evidence until agreement was reached on the appropriate score. Final results were then integrated into the Vital Signs Profile, which was presented to the Minister of Health or equivalent for approval to be released. The typical length of time to complete the entire PHC Progression Model assessment process was approximately 3 months.
Findings and lessons learnt
The PHC Progression Model is a novel tool for systematically assessing PHC capacity at a national level and provides a basis for countries to track their progress in creating better conditions for stronger performance over time. Additionally, due to its standardised methodology and structure, the PHC Progression Model and the Vital Signs Profile overall enables countries to engage in cross-country learning and peer-to-peer benchmarking if they choose, though neither tool is intended to be used for direct ranking or comparison purposes.
PHCPI conducted targeted outreach with implementers of the first five pilot assessments to collect their insights on the process and lessons learnt. Overall, the results of the five PHC Progression Model assessments (table 2) demonstrate that the process and methodology were feasible and acceptable. The measurement tool was able to be implemented with fidelity and to detect meaningful variation in PHC system capacity across and within countries. The tool also had internal validity, with internal and external scores being highly aligned—on average across all countries, 60% of measure scores were fully aligned across internal, external, and consensus scores while only 7% of scores differed by two or more performance levels between internal and final consensus scores. Only one measure had two or more countries differ by at least two performance levels from the internal to final consensus score. Importantly, not all differences between the internal and external scores shown in table 2 were due to PHCPI ‘correcting’ internal scores; in cases with misalignment, external scores were both higher and lower than internal scores, and it was typical that discussions about the difference between internal and external scores would surface implicit knowledge being applied by in-country teams and lead to the identification of additional data sources and evidence to justify the initial scores and resolve discrepancies.
Overall, we found that completion of the PHC Progression Model assessment was a process which generated valuable new collaborations and insights for countries. The information needed to score individual measures and complete the assessment was often located within a multitude of documents and key informants. Bringing together all of these different data sources created a unique opportunity to collaboratively and holistically assess and understand an individual country’s PHC capacities in a way that is difficult if not impossible to do otherwise. This process of conducting the assessment and strategy of bringing together diverse stakeholders, each of whom had deep insight into a different piece of PHC, was often as valuable for understanding PHC capacity as the actual assessment results. Stakeholders reported that the process of implementing assessments made the identification of a system’s capacity strengths and weaknesses ‘glaringly obvious’ and that the assessment process resulted in learnings that ‘challenged pre-existing expectations’, even for stakeholders who had long been deeply embedded in the system. Since completion of assessments, all five country teams have initiated efforts to use the results of the PHC Progression Model and Vital Signs Profile to identify targeted improvement plans and inform efforts to expand data availability to more routinely measure areas assessed by the tool.
Our results also demonstrate that there are multiple implementation approaches that can be successfully employed to complete a PHC Progression Model assessment that is appropriate for the country context. Key core processes shared across countries that enabled success are summarised in box 3 and included contextualising the assessment within ongoing efforts by the government to improve PHC, obtaining both high-level ministerial buy-in and deep technical engagement, and customising the assessment strategy to meet local expectations and norms.
Implementation steps that enabled success across countries
Positioned within an ongoing strategic effort of the government to improve primary health care so that the assessment is not perceived as an ad hoc or standalone effort.
Obtaining high-level buy-in and leadership from the ministerial level.
Careful messaging of the assessment’s purpose as the first step in an improvement effort rather than as a punitive or audit tool.
Fostering a participatory assessment process.
Customising the assessment strategy—including working group composition, data collection strategies, and approach to the scoring exercise—to meet country expectations and norms.
Deep engagement by in-country technical teams with ongoing, trusting relationships with government officials.
Strong relationships between Primary Health Care Performance Initiative and the local assessment teams that established trust, mutual respect, and transparency.
The piloting of the PHC Progression Model identified limitations of the tool and assessment process. Most notably, the assessment was challenging to implement in a federalised country (Argentina) where the high degree of decentralisation meant that national-level data sources were unable to yield sufficient, timely information about on-the-ground realities across provinces. In federalised countries, subnational rather than national assessments of PHC capacity may be both more feasible to conduct and yield more informative results. Additionally, as noted above, completion of an assessment required investment of resources, including focused time from stakeholders and often necessitated hiring a consultant to support in-country efforts. We anticipate that repeat assessments in a country will be able to build on results and infrastructure established through the initial assessment, and therefore require less time and fewer resources. However, it will be critical for PHCPI to identify ways to streamline the assessment process and increase sustainability of the PHC Progression Model. Finally, the external validation process was designed to ensure that the collected data supported the scores proposed by country teams, however, external teams did not conduct quality reviews of data sources or any independent data collection efforts.
Results of each country’s assessment were released as part of their complete Vital Signs Profile at the Global Conference on PHC in Astana, Kazakhstan in 2018. Following the successful piloting of the PHC Progression Model, PHCPI undertook efforts—including expert consultations and a convening of early implementers—to refine the assessment tool to address any challenges identified during the pilot phase and ensure the measurement criteria in the tool reflect guidance and standards released since tool development began in 2017. In April 2019, PHCPI released an updated version of the assessment tool.23 Additionally, PHCPI updated the assessment guide to incorporate lessons learnt from the pilot experience; more information about the assessment guide is available on request.22 Moving forward, PHCPI has already initiated partnerships with ten additional countries—with many more planned—to expand the use of this assessment.
Through a structured, participatory process, PHCPI developed and piloted a novel assessment tool to systematically measure PHC system capacity. Pilot assessments in five countries found the tool to be feasible to implement with fidelity, acceptable to stakeholders, and able to produce valid results. Both the assessment process and results were found to be highly valuable to country stakeholders and are now being used to inform improvement efforts. The PHC Progression Model is a promising new approach for generating comprehensive, standardised, and actionable data on PHC capacity to complement other key performance indicators and develop a holistic understanding of PHC strengths and weaknesses that can be used to drive improvement efforts.
The authors gratefully acknowledge the contributions to the development and implementation of the PHC Progression Model made by the entire PHCPI working group and in particular by Beth Tritter and Hong Wang. We also thank Etienne Langlois and Abdul Ghaffar from the Alliance for Health Policy and Systems Research and all those who participated in our April 2018 expert convening for their time and feedback during the development process. Finally, we are immensely grateful to all individuals who supported the implementation of PHC Progression Model assessments in Argentina, Ghana, Rwanda, Senegal, and Tanzania.
Handling editor Seye Abimbola
Contributors HLR, DS, LRH and AB led the planning of the work described here. All authors contributed to the conduct of this research. HLR drafted the manuscript and all authors provided review and revisions. All authors gave their approval for publication.
Funding Funding for this work was provided by the Bill & Melinda Gates Foundation. Foundation staff contributed to the implementation of the work described here and one funder representative is included as a coauthor.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No additional data are available.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.