A new way of measuring PHC capacity

To identify a means of systematically assessing PHC capacity, PHCPI built on the recent increase in use of mixed-methods, rubric-based assessment tools in the health landscape. These tools are designed to capture performance across a range of different levels of system maturity, as defined by prespecified performance categories described in a series of rubrics. As a first step in conceptualising the PHC Progression Model, we used literature reviews, internet searches, expert recommendations, and snowball sampling to search for rubric-based tools in the healthcare domain. We identified a multitude of relevant tools, and six in particular that substantially influenced the conceptualisation and design of our work.^9–14 Through a review of publicly available materials and interviews with the developers and/or implementers of these tools, we extracted relevant key learnings and the best practices around tool design and implementation (box 2). Additional information about these tools is available in online supplementary file 1.

Based on these lessons learnt, PHCPI decided that the new assessment tool should be implemented through a joint internal and external assessment process led by in-country teams who rigorously document findings to collectively determine scores that are subsequently reviewed by an external team to ensure that results are rooted in evidence and that performance standards are consistently applied across countries. Such a process offered the best means of achieving PHCPI’s two goals for the assessment: (1) to produce a national assessment of PHC capacity that is acceptable to and owned by policymakers in LMIC and used to drive improvement efforts and (2) to drive accountability and improvement through the public release of Vital Signs Profiles that are standardised across countries and enable peer-to-peer benchmarking and learning.

Development of the tool

PHCPI undertook a structured, iterative, and participatory process to develop the PHC Progression Model (figure 2). The first step was a targeted review of relevant global health frameworks, toolkits, and data collection instruments15–20 to identify any indicators or normative standards related to the content to be assessed via the PHC Progression Model. Drawing where possible from these validated tools and supplementing with qualitative indicators to create a set of rubrics outlining four progressive performance categories for each topic assessed, we completed a first draft of the PHC Progression Model. To assess the face validity of the tool, we then partnered with the Alliance for Health Policy and Systems Research at the World Health Organization (WHO) to complete mock assessments using the seven Primary Health Care Systems case studies that were available at the time (Bangladesh, Ethiopia, Ghana, Kenya, Nigeria, Tanzania and Uganda).21 Based on the findings of this exercise, we made adjustments and generated a second draft of the tool.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Timeline of the development of the PHC Progression Model. LMIC, low-income and middle-income country; PHC, primary healthcare; PHCPI, Primary Health Care Performance Initiative.

Next, we identified 75 individuals with expertise in specific areas covered by the PHC Progression Model and conducted key informant interviews and online surveys. Each expert was asked to review the sections of the tool relevant to their area of expertise and reflect on the following dimensions of each measure: relevance, calibration, reliability, and comprehensiveness.

Finally, we convened an expert advisory group made up of international measurement experts as well as policy-makers from seven LMICs to review the content of the PHC Progression Model and assess whether each measure included in the tool captured the right information, was clear and well calibrated, and was feasible to assess in the represented countries. Based on all expert input received, we updated the PHC Progression Model to a version ready for pilot testing.

The PHC Progression Model

The resulting PHC Progression Model is a mixed-methods assessment tool for measuring the foundational capacities of PHC.22 The model consists of 32 measures covering the content areas described in box 1. As shown in figure 3, each measure includes a rubric that is used to assign a country to one of four performance categories ranging from level 1 (low) to level 4 (high). The criteria for levels 1–4 vary according to the content of each measure being assessed; individual criteria can be found in the assessment tool.22 Data and evidence for completing the assessment can be drawn from a variety of sources available within countries, including policy documents; routine reports and assessments; data elements from global surveys and/or locally owned and generated data; and key informant interviews with a variety of public and private, governmental and non-governmental organisations including at the subnational levels, as appropriate depending on the measure to ensure inclusion of diverse perspectives.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Structure of the PHC Progression Model and its relationship to the Vital Signs Profile (VSP). Each of the 32 measures of the PHC Progression Model contains a rubric outlining four performance categories (Levels 1–4). Measures are grouped thematically, according to the PHCPI conceptual framework. Raw measure scores are averaged by theme into nine subscores, which are in turn averaged to calculate the three scores that appear in the Capacity pillar of the VSP. Subscores and VSP scores are rounded to the tenths place. PHC, primary healthcare; PHCPI, Primary Health Care Performance Initiative.

Scoring of each measure of the PHC Progression Model employs a threshold approach, in which a performance level can only be achieved if all criteria described in the measure meet the performance described in the corresponding rubric. The 32 measure scores are summarised into nine subscores, corresponding to each subdomain of the PHCPI conceptual framework being assessed, by taking a simple unweighted average of all the constituent measures. The nine subscores are then summarised into three overall scores—Governance, Inputs, and Population Health and Facility Management—for display in the Capacity pillar of the Vital Signs Profile, again by taking simple unweighted average of the constituent subscores (figure 3).

Piloting of the PHC Progression Model

PHCPI partnered with national Ministries of Health in five LMICs—Argentina, Ghana, Rwanda, Senegal, and the United Republic of Tanzania—to pilot a PHC Progression Model assessment in 2018. Countries were selected based on a formal expression of interest by the minister on behalf of their government, strong pre-existing relationship between one of the PHPCI partners and the ministry, and availability of resources to support the assessment.

Each country formed a core team responsible for implementing the assessment in accordance with the methods and standards outlined in a standardised assessment guide. As described in table 1, the size and composition of the core team varied by country depending on factors such as the time and availability of technical staff and preferences for inclusivity across multiple organisations or divisions within the Ministry of Health. Most often, the core team contained technical staff from Ministries of Health with expertise in and oversight of PHC in their country, as well as technical consultants who were often responsible for data collection.

The assessment team began by contextualising the assessment to the country context, a process which entailed agreeing to local definitions for: (1) the package of services considered to make up ‘PHC’, (2) the facilities considered to be ‘PHC facilities’ and (3) the human resources for health considered to be ‘PHC human resources’. Next, country teams undertook a detailed review of the types of data that would be required to score each measure, identified potential quantitative and qualitative sources, and made a plan for efficiently collecting needed data. When key informant interviews at the subnational level were deemed necessary, country teams also developed a sampling strategy to ensure that subnational data sources would yield a representative picture of what was truly occurring in the country.

Countries approached data collection in diverse ways, with some choosing to centralise the process within one or two individuals and others distributing responsibility to multiple members of the assessment team. Teams based decisions around considerations of feasibility, acceptability and effectiveness, including: how to structure their assessment process based on experience with similar assessment methods, for example, the Joint External Evaluation12; the time and resources key stakeholders could commit to the process; and an understanding of what steps would be necessary to generate local ownership of the results and be most likely to encourage the use of results to drive improvement efforts. Table 1 summarises the different assessment processes undertaken by each country team.

Data collection plans were approved by stakeholders such as high-level representatives from Ministries of Health and core implementing partners whose acceptance of the results would be critical for ensuring resulting data would be used to drive improvement efforts. Teams then completed data collection and used templates provided by the PHCPI team to compile and synthesise relevant information from across all data sources for each measure.

These data syntheses were then used as the evidence base for an internal scoring exercise, which consisted of convening a stakeholder group to review all of the assembled evidence for each measure and use the rubric to assign the country’s performance to one of the four performance categories. In the event that assessment teams had been unable to identify sufficient data to score a measure, a score of level 1 was assigned. (In Argentina, the federal administrative nature of the health system and current events made assessment of Population Health and Facility Management unfeasible, and results were instead displayed as ‘N/A’.) Participants in the scoring exercise were selected by the assessment team based on how best to generate buy-in for and acceptance of the results.

Next, the results of the internal scoring exercise, along with all supporting evidence, were shared with the PHCPI country engagement lead and a team from PHCPI partner organisation Ariadne Labs for external validation. The goals of the external validation were to ensure that the available evidence justified the scores given by the country team and that measurement standards were being consistently applied across countries. Often, the external assessment process resulted in the identification of measures where more detailed evidence was needed to justify the internal scores; in these cases, the external and country assessment teams would engage in ongoing dialogue and review of additional evidence until agreement was reached on the appropriate score. Final results were then integrated into the Vital Signs Profile, which was presented to the Minister of Health or equivalent for approval to be released. The typical length of time to complete the entire PHC Progression Model assessment process was approximately 3 months.

Findings and lessons learnt

The PHC Progression Model is a novel tool for systematically assessing PHC capacity at a national level and provides a basis for countries to track their progress in creating better conditions for stronger performance over time. Additionally, due to its standardised methodology and structure, the PHC Progression Model and the Vital Signs Profile overall enables countries to engage in cross-country learning and peer-to-peer benchmarking if they choose, though neither tool is intended to be used for direct ranking or comparison purposes.

PHCPI conducted targeted outreach with implementers of the first five pilot assessments to collect their insights on the process and lessons learnt. Overall, the results of the five PHC Progression Model assessments (table 2) demonstrate that the process and methodology were feasible and acceptable. The measurement tool was able to be implemented with fidelity and to detect meaningful variation in PHC system capacity across and within countries. The tool also had internal validity, with internal and external scores being highly aligned—on average across all countries, 60% of measure scores were fully aligned across internal, external, and consensus scores while only 7% of scores differed by two or more performance levels between internal and final consensus scores. Only one measure had two or more countries differ by at least two performance levels from the internal to final consensus score. Importantly, not all differences between the internal and external scores shown in table 2 were due to PHCPI ‘correcting’ internal scores; in cases with misalignment, external scores were both higher and lower than internal scores, and it was typical that discussions about the difference between internal and external scores would surface implicit knowledge being applied by in-country teams and lead to the identification of additional data sources and evidence to justify the initial scores and resolve discrepancies.

Overall, we found that completion of the PHC Progression Model assessment was a process which generated valuable new collaborations and insights for countries. The information needed to score individual measures and complete the assessment was often located within a multitude of documents and key informants. Bringing together all of these different data sources created a unique opportunity to collaboratively and holistically assess and understand an individual country’s PHC capacities in a way that is difficult if not impossible to do otherwise. This process of conducting the assessment and strategy of bringing together diverse stakeholders, each of whom had deep insight into a different piece of PHC, was often as valuable for understanding PHC capacity as the actual assessment results. Stakeholders reported that the process of implementing assessments made the identification of a system’s capacity strengths and weaknesses ‘glaringly obvious’ and that the assessment process resulted in learnings that ‘challenged pre-existing expectations’, even for stakeholders who had long been deeply embedded in the system. Since completion of assessments, all five country teams have initiated efforts to use the results of the PHC Progression Model and Vital Signs Profile to identify targeted improvement plans and inform efforts to expand data availability to more routinely measure areas assessed by the tool.

Our results also demonstrate that there are multiple implementation approaches that can be successfully employed to complete a PHC Progression Model assessment that is appropriate for the country context. Key core processes shared across countries that enabled success are summarised in box 3 and included contextualising the assessment within ongoing efforts by the government to improve PHC, obtaining both high-level ministerial buy-in and deep technical engagement, and customising the assessment strategy to meet local expectations and norms.

The piloting of the PHC Progression Model identified limitations of the tool and assessment process. Most notably, the assessment was challenging to implement in a federalised country (Argentina) where the high degree of decentralisation meant that national-level data sources were unable to yield sufficient, timely information about on-the-ground realities across provinces. In federalised countries, subnational rather than national assessments of PHC capacity may be both more feasible to conduct and yield more informative results. Additionally, as noted above, completion of an assessment required investment of resources, including focused time from stakeholders and often necessitated hiring a consultant to support in-country efforts. We anticipate that repeat assessments in a country will be able to build on results and infrastructure established through the initial assessment, and therefore require less time and fewer resources. However, it will be critical for PHCPI to identify ways to streamline the assessment process and increase sustainability of the PHC Progression Model. Finally, the external validation process was designed to ensure that the collected data supported the scores proposed by country teams, however, external teams did not conduct quality reviews of data sources or any independent data collection efforts.

Future directions

Results of each country’s assessment were released as part of their complete Vital Signs Profile at the Global Conference on PHC in Astana, Kazakhstan in 2018. Following the successful piloting of the PHC Progression Model, PHCPI undertook efforts—including expert consultations and a convening of early implementers—to refine the assessment tool to address any challenges identified during the pilot phase and ensure the measurement criteria in the tool reflect guidance and standards released since tool development began in 2017. In April 2019, PHCPI released an updated version of the assessment tool.23 Additionally, PHCPI updated the assessment guide to incorporate lessons learnt from the pilot experience; more information about the assessment guide is available on request.22 Moving forward, PHCPI has already initiated partnerships with ten additional countries—with many more planned—to expand the use of this assessment.