Evidence gaps in effective health service delivery in low/middle-income countries (LMICs)

There has been a lot of international effort to generate evidence to inform the choice of cost-effective interventions for LMICs.10–14 A number of studies have shown that the impact of these interventions depends on the presence of a functioning health system and adequate resources.14 The health systems of many LMICs resemble complex adaptive systems,5 15 16 in which it is essential to adapt an intervention to the local context.14 This is especially the case with mixed health systems with a variety of service providers (public and private) and a regulatory system in which the behaviour of different actors may be influenced by multiple factors. A review of effective health service delivery in LMICs by the World Bank has found that many well-designed intervention programmes may not necessarily lead to expected outcomes.14 The key health reforms in China are an example, where despite detailed design, interventions, such as the essential drugs policy, did not achieve the expected outcomes at its initial stage and even led to unexpected negative consequences.5

Deliberation-based learning in China

In a rapidly changing context, the ability of a health system to adapt is crucial to maintaining sound and rapid development. Learning health systems that build on available knowledge and continue to learn from errors are better able to adapt to a changing environment and innovate successfully.17 Such learning health systems often adopt an experimentalist style of policy-making and develop a capacity to assess and scale up innovations.

Many analysts have identified factors other than the availability of evidence that influence health decision-making process.18–20 Greenhalgh and Russell19 argue for a constructive approach to using rhetorical argumentation for a ‘fair’ policymaking process. Decisions are considered as context-sensitive choices made through negotiation and deliberation in uncertainty. Such an argumentation or deliberative process is especially necessary in making difficult health decisions such as resource allocation and rationing of health services.

In such deliberation-based learning, key stakeholders are engaged in processes of dialogue and reflection, which link past to future actions. This deliberation approach is considered to contextualise problems and develop a collective understanding and consensus on how to anticipate, prevent and solve potential problems.17 21 22

China has made extensive use of experimentation and rapid assessment of innovations in its management of reforms, including in the health system. As in other sectors, reform of the health system has been through a process of trial and error, including subnational experimentation, innovation and learning.8 21 23 Some key Chinese think tanks have experienced rapid development and adopted and adapted new methods and approaches for evaluating policy innovations, and promoting learning from pilot reforms.5 8 Deliberations are widely employed in the design and planning stages of large pilot programmes,8 21 for better understanding the context and tailoring local interventions.

Building a learning platform

In early 2016, while the NHC and UNICEF were designing the project, CNHDRC was invited to design the monitoring and evaluation component. It puts together a team composed of national and local experts. It then convened a meeting for all participating counties and provinces to discuss the programme and the monitoring and evaluation (M&E) plan.

The situation analyses were re-designed on the basis of feedback from local researchers, decision-makers and project implementors. In addition to cost-effectiveness analysis and bottleneck analysis, other elements were added, including an analysis of the local socioeconomic and ethnographic situation and interviews with caregivers and receivers. Working together with local stakeholders, the CNHDRC undertook an assessment of the local health services, which served as the basis on which interventions would later be designed. This approach has proven to be effective in contextualising the pilot and its interventions, and in reaching consensus regarding approaches to local implementation.

Planning and design of the interventions took place in late 2016 and in 2017, during which time CNHDRC engaged 20 counties in selecting interventions, setting the expected objectives and understanding key implementation issues. OHT is a data hungry tool and requires rich data inputs for making accurate estimates. The research team worked closely with local M&E experts as well as implementors in pooling the data. The collaboration process itself served as a learning opportunity, through which local and national stakeholders improved their communication regarding the pilot design and M&E plan.

Deliberations before, during and after the research

Deliberations were organised before the design of the situation analysis was finalised, to enable local stakeholders to contribute to the overall design. In March 2017, the CNHDRC convened an inception workshop in Beijing that was attended by 35 people from four provinces and 20 pilot counties. Participants included decision-makers from the Provincial Health Commissions plus prospective implementers from county health bureaus and county MCH hospitals. The agenda was designed to inform participants about the overall objectives and the planned situation analyses, as well as to gather information on the counties’ expectations of the pilot—the problems the project would help the counties to solve—and the support they would need to achieve success.

The workshop showed that the cost-effectiveness of the interventions was only one consideration for the counties. Stakeholders said that they would need support from provincial officials to help them work through local challenges and gain access to resources. Overall, 33 of 35 attendees recognised that the pilot was in line with national and provincial policy priorities, although 25 thought it would be difficult to implement, due to resource and skill constraints. This meeting was a turning point for the design of the situation analyses. The researchers decided to include more county-tailored inputs besides cost-effectiveness and bottleneck analysis—such as context analysis and health system assessment—to help ensure successful implementation.

Deliberations were organised along with data collection activities, such as field surveys, to facilitate learning between local, provincial and national stakeholders, as well as to secure buy-in from key policy actors. Between June and August 2017, field visits to rural counties and provinces were conducted, which usually began with meetings with the Provincial Health Commission and other key agencies in the provincial capital to discuss the project’s aims, to secure the province’s support and to gather information on relevant local policies and initiatives. The team brought together a wide range of experts including CNHDRC staff, representatives of the Chinese Centre for Disease Control and Prevention (CDC), national-level clinical experts and local university-based researchers. The local project management agency was the provincial MCH Hospital, which has an overview of the MCH situation.

Following provincial meetings, the team—including central and provincial representatives and local university staff—visited the counties. This broad representation communicated the importance of the pilot and helped to build a common understanding of its aims, the challenges faced by the counties and their support needs. Information was collected through questionnaires, stakeholder interviews, examination of records and facility data and observation in facilities. In this way, the team gained a good understanding of the situation in the counties, including capacity levels and readiness for the pilot.

Deliberations were used to communicate study findings and help the local stakeholders make implementation plans. In August 2017, the NHC and UNICEF organised a project launch meeting. This was a one and a half days’ event, in which the CNHDRC presented the findings of the situation analyses in the morning session and parallel group discussions were scheduled in the afternoon to engage local and provincial stakeholders in discussing research findings and reflecting on their implementation plans. The M&E experts partnered with technical experts and facilitated the group discussion as follows: (1) presentation of a summary of the current maternal and child health and health services in each county and the province, including potential benefits and bottlenecks of implementing the pilot, and recommendations for overcoming the issues; (2) comments on the findings and reflections on the potential implementation plan and (3) drafting the implementation plan with troubleshooting and consultation of technical and management issues by the CDC expert, clinician and project coordination staff. In the morning session the following day, representatives of the participating counties and provinces were invited to present their draft action plans, and policy-makers, experts and the CNHDRC researchers made comments. In an evaluation of the meeting, people from the pilot sites claimed that they benefited from this new way of ‘learning and planning together’. Further meetings after the launch helped the counties to continue to receive feedback and refine their implementation plans.

Multilevel learning and relevant outcomes

The deliberations in the SNP project, mostly in the form of research-related meetings and field visits, facilitated many kinds of learning by key stakeholders at different levels.

The pre-launch discussions enabled the national pilot programme to take account of local needs and context-specific considerations. Learning on the part of the national policymakers and experts (including CNHDRC staff) resulted in a re-design of the situation analyses and the adoption of a more locally oriented approach to the pilot study.

Site visits and meetings during the situation analyses helped the CNHDRC team to spot potential barriers to implementation, such as insufficient staffing and low capacity, and also enabled counties to articulate clearer and more targeted requests for support. This helped central stakeholders to better understand the capacities of the counties and the areas in which they would need assistance.

In the major deliberation event—the project launch meeting, the sharing of data and evidence, and the discussions of opportunities and bottlenecks helped the provinces and counties design more realistic implementation plans. This way of organising the event combined three objectives: (1) managed discussion, reflections and consultation on the national pilot programme and its potential implementation in different settings, to support all the stakeholders to rethink programme design and implementation based on currently available best evidence; (2) facilitating direct and open communication between local, provincial and national policy-makers and implementing organisations, to support consensus on key issues concerning the national pilot implementation, such as who should do what and with what available resources; (3) providing targeted support to the local pilot counties and provinces in designing a realistic implementation plan, based on realistic situational analysis and extra onsite consultation from national and local experts. Such deliberation is complex and difficult to manage. CNHDRC, as a national think tank with a legitimate role in supplying evidence and knowledge into the health policymaking process, led the design of the important deliberation event.

By convening group discussions and troubleshooting sessions, the research team facilitated peer-to-peer learning between the province and its counties, integrated national expertise with local tacit or experiential knowledge, and linked the stakeholder’s past experience with future implementation plans. The clinical experts reported on the lack of proper skills and essential drugs and technologies for delivering the interventions. The CNHDRC and local university researchers discussed their worries about the anticipated workload with representatives from the county-level government and documented their concerns. Project officers at various levels communicated the management plan and tried to match it with the local conditions. All counties reported a lack of health staff, inadequate skills and competences, and poor access to essential medicines and medical technologies. Cultural and language issues were also reported as an important barrier to accessing essential neonatal care, as many ethnic minorities reside in China’s remote counties.

This kind of deliberation-based learning helped to assess barriers to success, and prepare all stakeholders for uncertainties and potential surprises. It surfaced richer messages and inputs than survey data and technical report writing. This extra consultation and communication has played a role in facilitating multilevel communication and understanding. For instance, in the meeting with Yuexi County in Sichuan Province, the team was told about the weak infrastructure there and about local concerns about lack of personnel for the pilot work. They also heard about failure of previous projects, and understood the situation by talking with frontline health workers. The county was finally withdrawn from the list of pilot sites as a result.

Deliberations also supported consolidated learning and articulating needs. For instance, the researcher–implementer mixed group discussions facilitated understanding and uptake of the study findings. Rather than circulating county-specific technical reports, the CNHDRC team presented key results in a policy-briefing style. Neonatal deaths, challenges to implementation, potential health benefits and costing of priority early essential newborn care (EENC) interventions were shown in figures and summary tables. Interactions between decisionmakers, researchers, managers and health workers during the launch meeting and particularly in the group discussions consolidated mutual learning and built trust and connection between multiple stakeholders. All participating parties got a chance to articulate their perspectives and interpretation explicitly.

As a result of the meetings, visits and collaborations described above, when the counties kicked off implementation in late 2017, they had much more detailed implementation plans than would otherwise have been possible. Most counties launched their programmes as expected and have chosen high-impact interventions that cover 90%–95% of the eligible population. Ningxia and Sichuan provinces are already rolling out this package of interventions province-wide. Implementation strategies in these provinces have drawn on the lessons learnt by the pilot counties to include training and capacity-building for medical staff and project managers, and to build an information reporting system to monitor progress.