The four guideline meetings 2005–2015

In 2005 the Cochrane Collaboration and the UK’s National Institute for Health and Care Excellence had been established for only 13 and 6 years, respectively, and the WHO had no specific procedures to incorporate research into the guideline development process table 1.4 Recognising the need for improved access to usable guidance,2 a small research team in Kenya was formed, endorsed by the Ministry of Health, with two main aims. First was to improve the use of research evidence in creating national clinical policies. This was manifest in 2005 as an ambitious aim to conduct systematic, contextualised evidence summaries for 14 clinical topics. The second was to engage government personnel and those with a major potential role in promoting ownership and adoption of policy in a shared decision-making process. The team initially trained itself in review methodology before engaging with an emerging, voluntary international grouping6 that subsequently went on to support the production of the WHO Pocketbook of Hospital Care for Children,7 to which the Kenyan group made contributions.

Initial reviews (eg, refs 8 9) were used to inform discussions at a first ‘Child Health Evidence Week’ held in June 2005. At this meeting 35 participants (table 1) spent an entire week briefly exploring and then debating the evidence before agreeing policy recommendations on the treatment of pneumonia, malaria, diarrhoea, meningitis, malnutrition, neonatal sepsis, feeding of the preterm and common complications of illness (see table 2). The participants also approved in draft format treatment algorithms that would become a 36-page booklet of guidelines, a simple formulary and charts to support paediatric prescribing of medicines, fluids and feeds. Importantly, the focus was to provide guidance not for the professional paediatric community but as a basic standard for acute care provided at the referral level by non-specialist clinicians. This booklet, published by government, was also a key tool used in a popular training course (ETAT+) that aimed to improve skills and promote guideline adoption.10 ETAT+ training was first used in 2006 as part of a multifaceted intervention strategy shown to improve district hospital care.11 It also became integrated into undergraduate and postgraduate paediatric training from 200812 in Kenya’s largest medical school helping to popularise the guidelines.

With increasing use of the guidelines and based on feedback from an emerging group of ETAT+ instructors, demand for their revision grew. The first revision was undertaken in 2010 (table 1). This incorporated for the first time in Kenya training in and use of the GRADE approach13 to inform evidence synthesis and culminated in a second Child Health Evidence Week. In 2010 non-research paediatricians were engaged in reviews and given basic training in review methodology and GRADE. At the guidelines meeting there was a greater focus on structured, facilitated discussion to contextualise evidence and develop draft recommendations that were voted on using a modified GRADE grid and individual, anonymous voting slips. Voting patterns were presented back to all participants used to create draft recommendations submitted to the Ministry of Health. Visitors from Uganda, Rwanda and Tanzania observed the meeting that was facilitated by a senior researcher. Aspects of the entire process were evaluated14 15 with the assistance of international experts. The output was a revised set of guidelines subsequently published and disseminated (table 1).

Efforts to revise guidelines in 2013 were prompted by publication in late 2011 of a large East African trial, with results that contradicted prevailing thinking on fluid management of very sick children.16 With little guidance forthcoming from WHO, guideline champions and ETAT+ instructors (now an important practice grouping)17 were frequently questioned on what was the best course of action. A local systematic review18 and a guideline meeting to include participants from Uganda and Rwanda, countries where Kenyan guidance was also informing practice, were planned. At the same time important questions were being raised by the paediatric community about the use of chlorhexidine to clean the umbilical cord and the risks and benefits of the emerging use of hydroxyurea to treat sickle cell disease in young children. Therefore, three systematic reviews were undertaken with prominent roles of personnel from the Ministry of Health and University of Nairobi supported by the research team18–20 and the Cochrane Infectious Diseases group.

Development of the approach

The process of translating this evidence into policy changed considerably in 2013 (table 1). Three topic-specific and more formally conducted guideline panels, each with 16–20 selected panellists, were convened at the annual national paediatric conference (table 1). Each panel met for a whole day after receiving the systematic review and the primary literature 4 weeks in advance of their meeting. For the first time panels formally documented disclosure of interests and their judgement about the level of certainty of effects, and used the DECIDE framework to explore the risks and benefits, feasibility and acceptability of possible recommendations21 until a consensus was reached. The participation, process, key discussion points and recommendations were documented, made publicly available online and were passed to the Ministry of Health. Immediate dissemination of key findings and draft recommendations occurred at the national conference, and policies were changed and guideline booklets disseminated before the end of 2013. (WHO recommendations on the subject of fluid management were not published until 2016).22

The model for guideline development in 2013 was used again in 2015. For childhood pneumonia uncertainty in the global evidence examined in 201023 had prompted conduct of a local pragmatic trial.24 With these new results a panel was constituted and developed draft recommendations for a substantial change in treatment policy before the trial results were published. In recognition of poor quality of care25 and persistently high neonatal mortality,26 three neonatal guideline panels were also constituted (for topics see table 1). Once again emerging use of new technology, continuous positive airway pressure for newborns, which was entering district hospital practice without review of benefits and harms, prompted development of local guidance.

Reflections

Initial efforts in 2005 at review of evidence and its translation to policy through discussion and building consensus (informally) had limitations by the standards emerging at the time.27 However, even in 2005 a rigorous and systematic approach to identifying literature was employed, with search strategies based on the PICO framework (population, intervention, control and outcome), although searches have largely been restricted to the English-language literature. Importantly evidence syntheses included observational studies where necessary given the lack of randomised trial evidence for many questions of relevance to low-income countries (LICs). The process was further enhanced from 2010 by employing specific software (GRADEpro) that documents and makes transparent the appraisal decisions made for each included article, information that underpins the final determination of certainty in evidence of effects. The guidelines created have influenced undergraduate and specialist training in Kenya (and elsewhere) from 2008 to date and resulted in a high proportion of graduating doctors in Kenya carrying knowledge of the guidelines to their workplace complementing other modes of dissemination.12 The emergent broad approach linking systematic review, engagement in evidence appraisal, consensus-based decision making, development of guideline booklets and an adult education-oriented interactive form of knowledge and skills training represent an overarching effort to translate evidence to policy and subsequently into practice, with evidence of some success.17 Trainers on skills courses have become in many ways a community of practice receptive to evidence-based policy change.12 The presence of observers from neighbouring countries in 2010 and again in 2013 helped support wider use of these evidence-informed policies and linked training in Uganda, Rwanda and other countries,12 28 extending this community and increasing impact.

Technical and procedural improvements were made over time (table 1). These included methodological developments in conduct of systematic reviews, GRADE training, declaration of interests, debate and agreement on the certainty in evidence of effects from a health system perspective, facilitation of discussion informed by the DECIDE framework to develop consensus-based recommendations, and use of formally constructed panels representing multiple constituencies. As this process evolved, so did engagement with and participation of local academic clinicians, the national paediatric association, health professions’ regulatory bodies, and importantly the Ministry of Health spanning multiple departments and specific secondment of personnel (table 1 and figure 1). An important element of the process, which was new to most involved, was a formal process of decision makers declaring any potential interests in the outcome of the guideline discussions. As many LIC academics rely on sponsorship to attend international conferences and as the private sector is becoming more prominent as a research funder in LIC, such disclosures are important. These together with a publicly available listing of panel members and a summary of their panel discussions are a critical element of good practice in guideline development supporting the transparency and credibility of final recommendations.

In 2010 specific efforts were made to evaluate preferences for or effects of different forms of presenting evidence from systematic reviews.14 15 These evaluations indicated very limited familiarity with evidence synthesis or critical appraisal of research literature among panellists prior to meetings. They also indicated a preference for contextualised narrative presentations of synthesised evidence and that limited time was spent reading reviews in preparation for panel meetings, emphasising the value of increasingly structured presentation and discussion of the available evidence.14 Despite these apparent limitations qualitative work suggested active and pertinent engagement with the evidence during discussions, diversity of opinions and an ability to accept a consensus. They also suggested an enthusiasm for participation, greater appreciation of the value and limitations of evidence, and an emerging sense of ownership of the process,15 perhaps linked to a sense that roles were not just to endorse global recommendations.23 In 2012, 60% of clinicians encountered in a survey of 22 hospitals had copies of the guidelines29 and there was evidence of substantial practice change.17

Limitations remain however. Currently mechanisms for patient engagement in the policy process and data on costs and formal cost-effectiveness analyses are largely absent in many LICs. We suspect, as in Kenya, that there is often no structured process for identifying local priority topics in LIC. Often (and appropriately) global issues may drive the process or as in Kenya it may be somewhat reactive, drawing on informal communication from policy makers, the agenda of technical partners or researchers. Despite these limitations the model that has evolved in Kenya uses internationally defined good practice, to the extent that is feasible, and is probably one of the most structured and participatory approaches to national guideline development in a LIC in Sub-Saharan Africa. The process has provided the platform for a complete cycle of decision making spanning a request for better evidence,23 conduct of a pragmatic trial24 and a new round of decision making. It has also helped foster wider awareness of how evidence can inform policy and the technical aspects of this process.15

It is not enough to make even well-informed decisions. These must be linked to a clear plan for disseminating guidance and ideally ensuring such new guidance can rapidly reach those providing and receiving health worker training.

Possible lessons for other LIC embarking on this journey are summarised in box 1. More generally we compare the evolution of our efforts with what is known about the science of using evidence to inform policy that proposes six major mechanisms that influence success.30 The approach we describe spans five, and our experience would suggest that countries consider all of these from the inception of their own efforts. These are building awareness of and a positive attitude towards evidence-informed decision making, providing communication of and access to the evidence, providing interaction between decision makers and researchers, supporting decision makers in developing the skills to make sense of the evidence, and influencing the structure and process of decision making.30 We also suggest that sustained effort to build a partnership spanning research, government, academia, professional associations and practitioners is important. Over a decade more than 120 people were engaged in the decision-making process and more than 25 contributed to the conduct of systematic reviews. Linkage with a training course and its expert trainers, some involved in guideline panels, has helped extend the community involved in the evidence translation process. Broadly we suggest this overarching approach has helped evidence translation by creating a social or professional norm among both decision makers and practitioners to use evidence, has been an effective strategy for awareness raising at scale and has helped reshape professional identity towards acceptance of common practice standards.30

In conclusion, our report contributes to a relatively small literature exploring the process of national guideline development in LIC.31–33 Over time a trade-off between greater attention to complying with international recommendations for translating evidence into policy that increases costs and limits the number of topics that can be tackled34 has become apparent. Clearly rigour and transparency are important. However, efficiencies can be gained by countries sharing systematic reviews so they do not need to be repeated, a clear aim of organisations such as the Cochrane Collaboration and WHO. This does not obviate the clear requirement for making country-relevant decisions locally to take into account context and national values and preferences and promote ownership. While global guidance provides a useful benchmark, this should inform but not replace such processes. Our experience suggests that long-term partnership may enable successful, timely and credible national guideline development. Sustaining such approaches and embedding them within local institutions should be considered a part of the health system strengthening agenda.