Research•

Sustainability of using the WHO surgical safety checklist: a mixed-methods longitudinal evaluation following a nationwide blended educational implementation strategy in Madagascar

Abstract

Background The WHO Surgical Safety Checklist reduces postoperative complications by up to 50% with the biggest gains in low-income and middle-income countries (LMICs). However in LMICs, checklist use is sporadic and widespread implementation has hitherto been unsuccessful. In 2015/2016, we partnered with the Madagascar Ministry of Health to undertake nationwide implementation of the checklist. We report a longitudinal evaluation of checklist use at 12–18 months postimplementation.

Methods Hospitals were identified from the original cohort using purposive sampling. Using a concurrent triangulation mixed-methods design, the primary outcome was self-reported checklist use. Secondary outcomes included use of basic safety processes, assessment of team behaviour, predictors of checklist use, impact on individuals and organisational culture and identification of barriers. Data were collected during 1-day hospital visits using validated questionnaires, WHO Behaviourally Adjusted Rating Scale (WHOBARS) assessment tool and focus groups and analysed using descriptive statistics, multivariate linear regression and thematic analysis.

Results 175 individuals from 14 hospitals participated. 74% reported sustained checklist use after 15 months. Mean WHOBARS scores were high, indicating good team engagement. Sustained checklist use was associated with an improved overall understanding of patient safety but not with WHOBARS, hospital size or surgical volume. 87% reported improved understanding of patient safety and 83% increased job satisfaction. Thematic analysis identified improvements in hospital culture (teamwork and communication, preparation and organisation, trust and confidence) and hospital practice (pulse oximetry, timing of antibiotic prophylaxis, introduction of a surgical count). Lack of time in an emergency and obstructive leadership were the greatest implementation barriers.

Conclusion 74% of participants reported sustained checklist use 12–18 months following nationwide implementation in Madagascar, with associated improvements in job satisfaction, culture and compliance with safety procedures. Further work is required to examine this implementation model in other countries.

Key questions

What is already known?

  • The WHO surgical safety checklist reduces mortality and morbidity but there is a lack of nationwide systematic implementation in low-income and middle-income countries (LMICs) hindering penetration.

  • A blended educational strategy for nationwide checklist implementation overcomes known barriers to implementation and early uptake.

What are the new findings?

  • High fidelity nationwide checklist use was observed 12–18 months after initial implementation.

  • Checklist use was of high fidelity with respect to procedural compliance and team behaviour and increased operating room workers understanding of patient safety and personal satisfaction with their work.

What do the new findings imply?

  • If this model is transferable to other countries, then the approach holds promise for widespread checklist scale up across LMICs.

Introduction

Worldwide, surgery is not as safe as it could be. Globally, approximately 313 million surgical operations are performed annually, with an estimated mortality rate of 5%–10% and complication rate of 3%–17%.1 2 While some of this excess mortality may be attributable to hospital deficits in human resources and infrastructure,3 surgical safety can be improved by simple measures. In 2008, the WHO second Global Patient Safety Challenge: Safe Surgery Saves Lives, launched the Surgical Safety Checklist.4 This checklist reduces mortality and morbidity after surgery by up to 50% with the biggest gains in low-income and middle-income countries (LMICs).5 Since the launch of the checklist, repeated studies have shown similar benefit across a wide variety of settings.6–13 However, poor implementation of the checklist can negate the positive effect.13 14 Therefore, the research question is no longer ‘does the checklist work?’ but ‘how to make the checklist work in a sustainable way?’

WHO checklist use is standard practice in a number of developed countries,15–18 but is uncommon in LMICs where the biggest gains occur. Attempts at widespread LMIC implementation have hitherto been unsuccessful19 and successful checklist implementation in LMICs is largely limited to small or single centre implementations without longitudinal evaluation of sustainability.20–24

This study is a longitudinal evaluation to determine the sustainability of WHO checklist use in Madagascar 12–18 months after nationwide implementation.

In collaboration with the Ministry of Health, we used a blended educational implementation strategy to design a 3-day multidisciplinary course to facilitate nationwide implementation of the WHO checklist in Madagascar.25 26 From September 2015 to March 2016 we trained 427 operating room workers from 21 hospitals and reported 78% checklist use at 3–4 months in 20 out of the original 21 study hospitals.26 Details of the course structure and implementation actors, actions, action targets, temporality, dose, outcome and justification are previously published.25 26

Our first research objective was to evaluate the extent of sustained checklist use in Madagascar at 12–18 months after a nationwide implementation. In the absence of longitudinal evaluations of checklist use in LMICs and known wide variation (39%–100%) in checklist use in a cross-sectional evaluation in England,27 we hypothesised that sustained checklist use would be moderate, that is, that sustained use would be reported by 50% or more of participants at 12–18 months in Madagascar. Our secondary research objectives were to assess the quality of checklist administration in terms of team behaviour, to evaluate the impact of checklist use on operating room staff, practice and culture and to identify predictors and barriers to implementation.

Methods

Study design

We used a concurrent triangulation mixed-methods design,28 as recommended for complex patient-safety research,29 to address our research objectives.

The study was carried out over a 4-week period in April 2017.

Hospital selection

Over a 4-week period in April 2017, we aimed to visit 14 out of the original 21 regional hospitals for a day at a time. Hospitals were selected partly using purposive sampling based on hospital size (large and small) and prior performance at 4 months (good and poor) of checklist utilisation in order to give a representative sample of the original hospitals25 26 and partly on a pragmatic basis to maximise the number of hospitals that could be visited during the allotted research team timescale for the visit. No hospital declined to participate in this study or the original training. Prior to the evaluation each hospital received a letter from the Ministry of Health detailing the date and purpose of the evaluation. Hospital Directors were then contacted by telephone to arrange the visit and invite operating room staff to voluntarily participate in the evaluation.

Participant selection

All participants were operating room staff at the selected hospitals and included surgeons, surgical assistants, anaesthesia providers, operating room nurses and other operating room aides. All were over 18 years of age and gave written consent to participate. No incentives or per diems were given and participation was entirely voluntary.

Evaluation team

The evaluation team consisted of 3–4 people including at all times, one board-certified anaesthesiologist and a local Malagasy non-specialist surgical doctor. At least one member of the evaluation team had also been a member of the original training team in 2015/2016. All members of the team were trained in the using the WHO Behaviourally Adjusted Rating Scale (WHOBARS)30 assessment tool and had experience in leading focus groups.

Outcome measures

Our primary outcome was checklist use measured by a self-reported questionnaire, developed for the LMIC setting prior to this study by our team.26

The secondary outcomes were as follows.

  1. The use of six basic safety steps5 including verification of patient identity and site of intervention, assessment of difficult intubation risk, evaluation of the risk of major blood loss, administration of antibiotics before skin incision, counting swabs and use of a pulse oximeter measured using self-reporting via completion of the study questionnaire.26

  2. Team behaviour during checklist administration was measured using a validated tool called WHOBARS.30 WHOBARS30 assesses team behaviour during checklist administration in five domains: setting the stage, team engagement, communication activation, communication of problem anticipation and communication of process completion. Each domain is graded on a scale from 1 to 7, with higher scores indicating superior team behaviour. Each part of the checklist (sign in, time out and sign out) is assessed using the five behaviour domains. Scores are combined and then averaged to give an overall WHOBARS score (range 1–7) and can also be reported individually to highlight areas requiring further training. WHOBARS was assessed either by watching real-life surgery in the operating room or through simulation if no surgery was taking place at the time of the research team’s hospital site visit.

  3. Examination of the associations of checklist utilisation with hospital size, surgical volume, participant gender and WHOBARS score, as determined by multivariate linear regression.

  4. The impact of sustained checklist use on operating room staff, hospital practice and organisational culture, measured using questionnaire and focus groups.25 26 The questionnaire was modified from that previously published25 by the addition of three questions:

    1. Did your personal understanding of patient safety increase or decrease?

    2. Did your personal satisfaction at work increase or decrease?

    3. Did your personal stress at work increase or decrease?

    Responses were made using five-point Likert scales. The questionnaire is shown in the online supplementary appendix 1.

  5. Barriers to sustained checklist implementation identified by focus groups with operating room personnel, which took place in French within each one of the study hospitals.

Data collection

Data were collected during a 1-day hospital visit and consisted of:

  1. Self-administered anonymous questionnaire (online supplementary appendix 1) with five-point Likert scale response format. The questionnaire was written in Malagasy and adapted from that used in our previous work.25 26

  2. WHOBARS assessments were made using direct observations of checklist administration in real-time in the operating room. If no surgery occurred during the evaluation visit, we used simulation instead to measure WHOBARS. During simulation participants were asked to adopt their usual professional role. Simulation scenarios are detailed in the online supplementary appendix 2.

  3. One focus group per hospital with all available operating room staff who gave consent to participate, using a prior reported focus group guide.25 All focus groups took place in the participants’ hospital, lasted 30–60 min and were conducted in French or English with Malagasy concurrent translation, depending on the preference of the participants. Each focus group had a facilitator who moderated the discussion and a primary and secondary scribe who took notes of the discussion.

Data analysis

Simple descriptive statistics were used to explore the primary outcome. For secondary outcomes, WHOBARS scores were analysed using descriptive statistics and predictors of checklist utilisation were determined using multivariate linear regression to examine the association between covariates and each outcome measure. The primary outcome of interest was checklist use determined from questionnaire responses. Covariates included hospital size, surgical volume, participant gender, WHOBARS score, increased understanding of patient safety, increased job satisfaction and reduction in work stress due to the training received. We created a hierarchical model to account for clustering at the hospital level. For multivariate linear regression, Likert scale responses were scored on a scale of 0–4 and single blank answers were replaced with the average response for that hospital. Data analyses were performed in Microsoft Excel and R V.3.0 (www.r-project.org) and p values of<0.05 were considered significant.

Open questionnaire responses and all focus group data were translated into English. The data were grouped by category or question in Excel and then manually analysed using thematic analysis.31 32 Inductive thematic analysis was used to analyse culture change. Important topics were identified and highlighted by one or both researchers and grouped into related themes. Deductive thematic analysis was used, based on the six key safety steps used to analyse changes in practice. No software was used for the qualitative analyses.

Results

Hospital and participant demographics

Median time from initial checklist training in a hospital to evaluation was 15 months (range 12–18 months; IQR 13–16 months).

One hundred seventy individuals from 14 hospitals were invited to participate and all gave written informed consent to participate in the evaluation. Of these, 17 were excluded from final analysis after consent was taken. There were 17 student paramedics who considered themselves part of the operating room team but were identified as new students to the hospital during the study time period and so could not reliably comment on use or impact of the checklist. Final analysis was performed on the remaining 158 participants (male 87; female 66; gender not recorded 5).

Details of the hospitals and participant demographics including comparison with the prior evaluation (December 2014–May 2015)26 are shown in table 1. Most participants (67%) had attended the original checklist training. No hospital had received additional checklist training in the interim period.

Table 1
|
Details of hospital and participant demographics

Questionnaire assessment of checklist use

ne hundred and fifty-eight questionnaires were returned. Nine were returned blank or with only one question answered and were excluded leaving 149 for final analysis. One hundred and ten out of 149 (74%) respondents indicated they were still using the checklist (49% ‘always in full’ and 25% ‘always in part’) 12–18 months after the initial training. Of the six basic safety processes, pulse oximetry and verifying the patient’s identity and type of surgery were the most frequently reported as ‘always’ completed (88% and 70%, respectively), and difficult intubation was the least frequent (47%). Details are given in table 2.

Table 2
|
Frequency of self-reported use of the checklist and the six basic safety processes.
|

WHOBARS assessment of staff engagement during checklist administration

Checklist use was evaluated using WHOBARS to observe real cases in the operating room in seven hospitals, with simulation used in the remaining seven hospitals because the operating rooms were not in use during the research visit.

Mean overall WHOBARS score was 5.2 (range 1.4–7; IQR 4–7). Details are shown in table 3.

Table 3
|
Mean hospital WHOBARS scores and method of evaluation

Predictors of checklist use

For multivariate linear regression analysis, single blank questionnaire responses were replaced with the average for that hospital. The analysis (table 4) showed that checklist use was not associated with WHOBARS score, participant gender, hospital size, surgical volume, improved job satisfaction or reduction in work stress but was associated with an improved understanding of patient safety.

Table 4
|
Multivariate linear regression results with checklist use as the dependent variable

Impact of checklist: personal well-being and understanding of patient safety

The impact of the checklist on participant’s organisation, communication, teamwork, understanding of patient safety, work satisfaction and work stress is shown in figure 1.

Figure 1
Request permissionrequest permission icon
Figure 1

Impact of checklist use on teamwork, communication, organisation, personal understanding of patient safety, work satisfaction and work stress. Responses given as percentage of participants.

Impact of the checklist: hospital culture

On thematic analysis of the focus group responses, three key themes relating to change in hospital culture were reported: teamwork and communication; preparation and organisation and trust and confidence in each other. These themes were supported by the survey questionnaire responses as shown in figure 1.

Improved teamwork and communication

Nine out of 14 hospitals reported improved teamwork, better or easier communication and a more cohesive approach to patient management. This is demonstrated by the following quotes recorded from focus groups:

  • One of the most important things we learnt was how to work as a team (Hospital F).

  • We have more communication and easier communication within the team (Hospital L).

  • We have always had good rapport with each other although the checklist does give everyone the right to speak if they are unsure about something (Hospital H).

The focus group data were supported by questionnaire responses in which 91% (135/149) and 89% (132/149) of participants reported improved teamwork and communication, respectively (figure 1).

Improved preparation and organisation

Eight out of 14 hospitals reported improvement in the general organisation and preparation of the operating room before starting a case, as demonstrated in the following quotes:

  • We now have a reminder of what we need to do and are focused on the most important points (Hospital G).

  • There is better preparation of equipment before the operation starts (Hospital J).

  • Preparation for blood loss before it happens (Hospital D).

These quotes endorse the 90% of survey participants who reported improvements in hospital organisation.

Improved trust and confidence and reduction in stress

Seven out of 14 hospitals reported improved levels of trust and confidence and a reduction in stress within the operating room team. This also resulted in improved doctor–patient relationships as seen by the following recorded comments:

  • More confidence within the team and more confidence between doctor and patient. For example, patients who have been treated in other hospitals without checklist have been very impressed when they see us using it (Hospital I).

  • The checklist can avoid surprises during the operation, there is more trust between doctor and patient, and everyone feels more secure (Hospital J).

  • There is better security for patients and staff and reduced stress because we communicate better…. making sure blood is available before the intervention start (Hospital E).

These quotes support survey data showing improved job satisfaction, reduction in work stress and increased understanding of patient safety. Quantification of the thematic analysis at a hospital level is shown in the online supplementary appendix 3.

Impact of the checklist: improved operating room safety practices

The three biggest changes in practice were increased use of pulse oximetry (8/14 hospitals), introduction of counting of surgical needles, swabs and instruments (8/14 hospitals) and better timing of antibiotic prophylaxis (6/14 hospitals). Further details are given in the online supplementary appendix 3. Two hospitals (L and M) related stories of how the pulse oximeter had alerted the clinical team to a desaturating patient, allowing the anaesthetist to take appropriate action. Hospital F self-reported their surgical site infection rate had dropped from 28% to 14%, which they attributed to the checklist ensuring antibiotic prophylaxis administration prior to skin incision. Hospital G had drawn, using a permanent marker pen, a table for surgical counting on the operating room wall tiles. For each case the nurse, used an erasable marker to write the count on the wall which allowed everyone to see the count and remain engaged with the process.

Barriers to checklist implementation

The most frequently reported barrier to checklist use (n=6/14 hospitals) was lack of time to perform the checklist in an emergency. Two hospitals (I and M) indicated that this was because there was not enough staff at nights and weekends. However, the most influential barrier to implementation was the presence of negatively dominant or obstructive people. Both Hospitals B and C (WHOBARS scores of 2.4 and 1.6, respectively) had senior doctors who were actively opposed to checklist use. In Hospital B, the Hospital Director and nurse anaesthetists were supportive and trying to encourage checklist use but were in conflict with the only anaesthesia senior doctor and only surgeon who refused to participate. The focus group at Hospital C demonstrated a good understanding of the checklist and a desire among nursing staff to use the checklist. However, they described a dominant surgeon who adamantly opposed the checklist leading to nurses being fearful of interrupting the surgeon. On observing surgery in Hospital C, the dominant surgeon and nurses fear of speaking except when spoken to could be seen. There was poor team engagement and minimal verbal communication except for essential communication such as requesting items when needed without forewarning.

Detrimental effects of checklist implementation

Twelve out of 14 hospitals reported no detrimental effects associated with checklist implementation. Two hospitals (B and C) both with very low WHOBARS scores reported detrimental effects. This can be seen by the following comments:

  • Running the checklist out loud would interrupt surgeons’ concentration…the checklist doesn’t apply to Malagasy culture, where it’s not appropriate to check that a more senior person has done their job. …. it would be questioning (their) competence (Hospital C).

  • There is conflict between the team and the hospital director.…some people are very opposed to the checklist and others want to implement it (Hospital B).

Discussion

In this paper, we present a longitudinal study of sustained checklist implementation in Madagascar—to our knowledge, this is one of the first longitudinal large-scale evaluations of checklist use in a LMIC setting to date. Twelve to 18 months after a 3-day training course, there was widespread use of the checklist with 74% of participants still using the checklist. Since the checklist has a dose-dependent effect, compliance rates of less than 100% can still result in improved patient outcomes.6 9 11 Our results are comparable to studies in high-income countries. In England, Russ et al 27 reported that on average only two-thirds of items on the checklist were verified, sign out was not completed in 39% and team members were absent in more than 40% of cases. In New Zealand, the average percentage of checklist items completed ranged from 40% to 69%, and the operating room team engagement was often incomplete.33

The WHO checklist encourages procedural compliance to basic safety processes and aims to improve operating room safety by improving teamwork and communication. Teamwork and communication are known to influence patient outcome,34–37 and it is these aspects of the checklist that are proposed explanations for the success of the checklist in reducing mortality and morbidity by almost 50%.38 Disengaged or cynical use of the checklist may be detrimental.14 Thus, evaluations of checklist implementation must measure both procedural compliance and team behaviour15 and fidelity of checklist use (ie, use of the checklist in the spirit and manner with which it was designed).13 WHOBARS is specifically designed to measure the behavioural aspects of checklist utilisation. In our study, 7 out of 11 hospitals had a high WHOBARS (>5.3 (75%)) and 2 out of 11 had WHOBARS <3.5 (50%). High WHOBARS suggests effective team behaviour and constructive engagement during checklist implementation. Checklist use did not correlate with hospital size, surgical volume, WHOBARS, increased personal satisfaction or reduced stress at work, but was associated with an improved understanding of patient safety. However, most participants personally reported in questionnaires that checklist use had increased their understanding of patient safety and personal satisfaction with their work (87% and 83%, respectively), as well as improving their teamwork, communication and organisation. One explanation for this may be that the checklist requires a team approach not an individual one. Therefore, even though checklist use has a positive individual impact, that is not enough to significantly influence checklist use overall. WHOBARS scores were calculated using a small sample size at each hospital (only 1–3 observations) and half were calculated based on simulation observations rather than real operating room scenarios. This may have compromised the validity of the WHOBARS measurements and weakened any predicted effect on checklist use. A post hoc sensitivity analysis with WHOBARS excluded did not substantially change the correlations of the other variables.

Participants reported improved teamwork and communication, better organisation and preparation and a greater trust and confidence in each other and with patients. This level of positive impact of checklist use on individuals may partially account for the sustained effects since the benefits are tangible at a personal level and that may provide motivation to overcome local challenges. The most commonly reported challenge (6/14 hospitals) was lack of staff during emergency surgery. To overcome this challenge, many participants described trying to do what they could, or starting the surgery and then catching up with items on the checklist in retrospect. Operating room staff persisted in trying to overcome the difficulties perhaps because they could perceive the benefits of checklist use both personally (83%) and for patient safety (87%). This contrasted with two hospitals where dominant and influential team members demonstrated a negative or cynical attitude to the checklist. In these hospitals, staff felt powerless to do anything and said that the checklist had had a negative effect. This reinforces reports that negative or cynical checklist use can be detrimental.14

In focus group discussion, pulse oximetry, counting needles, swabs and instruments and timing of antibiotic prophylaxis were reported as the biggest changes in practice, with most hospitals incorporating these procedures into routine practice. The use of pulse oximetry shows the largest sustained improvement in practice since over 50% of hospitals did not have a pulse oximeter at the start of the project,39 but received a donated pulse oximeter as part of checklist training.26 Counting was not performed prior to the training and was taught as part of the checklist course.26 At 4 months post-training, checklist use was associated with counting instruments but commonly reported difficulties with counting included inability to recall the names of instruments and a lack of personnel to do the counting. One explanation for the association of checklist use with counting (at 4 months) was that if participants were motivated enough to overcome the difficulties of learning the names of the instruments, they may be motivated enough to use the checklist. By contrast in this study at 12–18 months post-training, no one reported difficulties with recalling names of instruments presumably because they had persisted and now knew all the names. Also lack of personnel to perform the surgical count was only a challenge in emergency surgery at 12–18 months because systems and processes had been worked out to incorporate counting into routine daytime work without increasing the number of team members.

The follow-up rate at 12–18 months in our longitudinal study was 37% (158/427 originally trained), compared with 47% (183/427) at 4 months.26 This is higher than other LMIC surgical evaluation studies that have reported follow-up rates of 17%–44%40–48 and slightly lower than response rates of 38%–70% for surveys and self-reporting studies in high-income countries.49 50 LMICs studies generally have lower follow-up rates than high-income countries due to challenges such as non-functioning telephone numbers and email addresses, inadequate record-keeping, transport costs and difficulties accessing rural locations.

This study has a number of limitations. Checklist use at the procedural level was self-reported and may be open to subjective bias, recall bias, under-reporting in the hope of getting further training or over-reporting to create a falsely good impression (ie, social desirability bias). WHOBARS was observed only for one or two interventions in the operating room in half of the hospitals and in up to three simulations in the remainder. For WHOBARS, a sample size of 9 is recommended to show differences between hospitals, but we did not have time to complete nine observations. However, we did not aim to make comparisons between hospitals using WHOBARS but rather to use WHOBARS to assess the behavioural aspect as well as the procedural aspect of checklist utilisation. The focus groups were not recorded and transcribed verbatim due to resource constraints and may be open to subjective recorder bias. Focus groups were heterogeneous and therefore nurses and other staff may have been compromised by social pressure and a hierarchical authority culture from speaking openly in front of surgeons. From our observations, whether or not nurses were compromised from speaking out depended on the surgical team dynamics and the culture of the hospital. In some hospitals, nurses were very vocal and willing to speak out and became ‘checklist champions’ and but in others they were quieter and it was very difficult to engage them in the focus group discussion. Only two-thirds of the original hospital sites were visited and even though no hospital had received further checklist training in the interim, there may have been other factors in the interim that effected operating room procedures, practice and culture outside of our control or knowledge. Follow-up rate from the original training sample of 427 participants was only 37%, but this is comparable to surgical outcome studies in LMICs.40–48 We are unable to contextualise this further because we do not know what per cent of the total surgical staff this represents. As described elsewhere,25 during the original training, entire perioperative teams were asked to be present and hospitals did not schedule non-emergency surgery during the 3-day training. This resulted in the majority of the perioperative staff attending the training. Yet, for the 4-month follow-up and for this study, no surgeries were postponed and no participation incentives were offered, which may have reduced the follow-up rate. Further, when hospital directors made the initial contact with participants prior to our visit, they may have assumed only a few participants would be sufficient to report back for the group as a whole.

Our study also has a number of strengths. To the best of our knowledge, this is the first longitudinal study of national checklist implementation in a LMIC. The study was designed to test the hypothesis that rapid (using a 3-day course) nationwide checklist implementation is not only possible but also has a sustainable impact. We measured procedural compliance and team behaviours as part of the study and triangulated these results with qualitative data from focus groups to give insight into the personal impact of the checklist on staff satisfaction as well as organisational culture. Our future aims are to identify the specific implementation strategies and outcomes associated with success in order to inform national implementation plans elsewhere.

In conclusion, our study shows that 12–18 months after a 3-day training course administered to all the regional hospitals in Madagascar, 74% of participants were still using the checklist, 83% reported that checklist use improved their work satisfaction and 64% (7/11) of hospitals had WHOBARS >75% indicating very good team engagement and communication during checklist administration. An improved general understanding of patient safety was predictive of checklist use but hospital size and surgical volume were not. Further research is needed to evaluate this 3-day checklist course and blended educational implementation model in other countries.

Article metrics
Altmetricopen-url
Dimensionsopen-url