Current climate of outcomes reporting in global surgery

First, it is essential for us to acknowledge the role of outcomes and our reasons for focusing on them in our evaluation of accountability in global health. By definition, healthcare outcomes are intrinsic to quality of care and in general they are relatively free of bias. However, due to the time required from care to outcome, linking care to outcomes can be a challenge. In light of this, all three elements of the Donabedian quality triad—process, structure and outcomes—are essential in building a successful healthcare system framework and should be considered, reported and monitored when working to improve quality of healthcare. Processes and structure can serve as surrogates for outcomes when outcomes have significant lag time or rarity (eg, mortality) that makes it difficult to adequately power studies.5 Without adequate oversight or reporting, none of these elements are appropriately monitored. We will focus primarily on outcomes reporting in global surgical endeavours due to its centrality to quality and concrete nature, with the understanding that process and structure should fall under the purview of a supervising body as well.

We recognise that even in high-income countries (HICs) with established surgical systems, outcomes reporting is not mandatory, or even common. In the USA, the National Surgical Quality Improvement Program (NSQIP) currently has 582 domestic hospitals enrolled, of over 5500 hospitals in the country.6 As outcomes are integral to evaluating care, we do not ignore or minimise the need for improvement in this arena in HICs. Rather, we see an opportunity in the LMIC context to build this critical function into their systems from the outset. Similarly, the danger of the lack of accountability for surgeons overseas, especially in areas with less robust legal oversight, adds urgency to the call for required outcomes reporting for humanitarian or global surgical missions.

Currently, most data published from humanitarian missions are observational reports of care provided, with limited outcomes reporting. Examples include military hospital ships or forward surgical team missions,7–10 short-term trips with a singular focus such as inguinal hernia repair11 12 or cleft lip/palate repair,13 and larger volume reports from entities like Médecins Sans Frontières14 and the Cinterandes Foundation’s mobile surgical unit.15 The outcomes reporting in these studies is largely intraoperative morbidity/mortality, immediate postoperative morbidity (within 2–3 days), and rarely 30-day morbidity, with follow-up rates around 20%–30%. An exception is the work of White et al,16 who reported 7-year follow-up of morbidity, social impact and patient experiences of humanitarian mission surgery (maxillofacial, plastics and orthopaedic) using semistructured interviews and questionnaires. Of 174 patients identified via purposive sampling, 41% participated in the evaluation, and the results included 35% with persistent postoperative pain and 18% who had sought follow-up for a clinical complication. However, 73% of patients felt that they reaped social benefit from surgery and the patient satisfaction rate was 89%.

Recognising that safe surgery is a multidisciplinary effort, the G4 Alliance17 has called for a similar level of outcomes reporting from the four key areas of surgical, trauma, obstetric and anaesthesia care. As part of their work, the G4 Alliance called on the Member States of the WHO to prioritise this issue and ensure mechanisms to collect national surgical and anaesthesia indicators. Several other organisations have also collectively championed ‘safe and effective surgery’ as an essential component of primary health coverage.18 19 The World Federation of Societies of Anesthesiologists adopted the International Standards for a Safe Practice of Anesthesia in 1992, and revisions were ratified in 2008 and in 2010. They are recommended standards for anaesthesia professionals throughout the world, intended to provide guidance and assistance to anaesthesia professionals, their professional societies, hospital and facility administrators, and governments for improving and maintaining the quality and safety of anaesthesia care. As part of the WHO’s effort to improve patient safety, they advocate the ‘Patient Safety Pulse Oximetry’ project, which aims to improve the safety of operating rooms worldwide.20 They include indicators such as caesarean section rate, proportion of operating theatres with pulse oximetry, trauma mortality rate and perioperative mortality rate. A recent consensus recommendation for short-term missions in paediatric general surgery and perioperative care is clearly a step in the right direction to ensure safe, responsible and ethical conduct by HIC providers.21 We agree that these are good places to begin for evaluating and building nascent surgical systems in LMICs.

A few pioneering publications have spearheaded the effort towards more comprehensive outcomes reporting. The Royal Colleges in the UK and Ireland have initiated an advocacy group with the support of the WHO for curriculum development, quality assurance and assessment of young surgeons.22 Orthopaedic surgeons were able to achieve 81.9% follow-up rates of 127 patients over 5 years via coordination with a Peruvian general surgeon—notably, this study began 3 years after the establishment of the programme which has been serving the same city in Peru since 2004 and provides graduated (increasing) financial compensation to patients for returning for follow-up.23 Supervised teaching of surgical techniques to general practitioners in Malawi led to an increase in volume of general surgical cases by 89%, and outcomes of hernia operations done by these practitioners in peripheral hospitals were comparable with those performed in central hospitals.24 Marck et al 25 demonstrated the value of early outcome studies in two consecutive missions to Ethiopia. By comparing outcomes in 2007 with those from 2008 after implementation of quality improvement methods, they show two important results: a decrease in overall complications, but a sustained high rate of complications with complex procedures. Finally, Jindal et al 26 obtained data on 60 corneal transplant recipients in Guyana, demonstrating dramatic improvement in quality of life and aversion of 260 disability-adjusted life years (4.3 per patient). These examples highlight the ability to demonstrate impact and improve quality during humanitarian missions if outcome measures are monitored and reported. Shrime et al 27 completed a systematic review of charitable platforms in global surgery which also demonstrated many good examples of outcomes reporting, such as Venkatesh et al,28 who reported 94% visual acuity improvement after cataract surgery with an 88% 40-day follow-up rate. While this review was able to compare efficacy of platforms in global surgery based on such studies that appropriately published outcomes, Shrime et al noted that they had to exclude platforms such as individual surgeon organised missions and missions that aim to establish training programmes because of a lack of outcomes reporting.

Thus these cases are the vanguard, and not yet the norm. In some cases, outcomes are reported for us by social media and investigative journalism, raising further questions of accuracy, bias, sensationalism and helpfulness. What are the obstacles to outcomes reporting, and what mechanisms can be employed going forward to create a replicable framework for outcomes reporting in global health?

Identifying obstacles

Arguably, the most self-evident obstacle is the logistical difficulty of follow-up. As mentioned above, even in studies that attempted follow-up, rates were around 20% and rarely more than 30 days after surgery. Perhaps the most successful example was a follow-up attempt by Operation Smile, which included 6-month and 12-month postoperative appointments with pictures and independent surgeon reviews; this effort garnered a 37% follow-up rate.12 Obstacles include transportation issues, record-keeping difficulties, and lack of coordination between medical teams themselves and between teams and local healthcare professionals.29 As has previously been stated, ‘without follow-up, the consequences of complications for operated patients could be disastrous’.30

Similarly, by nature, many global health missions take place in conflict or disaster zones. Despite advances in assessment, coordination and delivery of humanitarian aid during the Haiti earthquake (2010), information on surgical outcomes and patient follow-up was poorly documented.31

Even in the absence of logistical difficulties, compliance is an issue, as is demonstrated by the low rates of outcomes reporting even in HICs. As mentioned above, only about 10% of US hospitals currently participate in NSQIP, the American College of Surgeons’ quality initiative. The US-based Anesthesia Quality Institute (AQI) has launched another reporting forum that, while growing, is still voluntary.32 The AQI has identified this same need for safe anaesthesia in LMICs and has created a separate registry specifically for use by NGOs, but to date only one publication has documented its use.33 As safe anaesthesia care is inextricable from safe surgery, compliance with anaesthesia outcomes reporting is an integral part of surgery outcomes reporting.

When social media and journalists fill the information void, results range from effective whistle-blowing to inaccurate sensationalism. It can be difficult to separate one from the other. An older but instructive example is the 1999 New York Times exposé of avoidable deaths of patients undergoing relatively simple procedures for repair of cleft lips and cleft palates.34 Among other accusations reported in the article, a paediatric surgeon in Bolivia branded the organisation’s practices ‘a form of neo-colonialism’; eventually, the reports led to structural changes in the involved charities. More recently, the online news site for the American Anthropological Association published an ethnographic research study on women in Niger presurgery and postsurgery for obstetric fistula.35 After 18 months of interviews of over 100 women, their data make a convincing argument that surgical outcomes statistics were sharply skewed and did not correlate with the results experienced by the patients. While such incongruity may be explicable by the difference between patient-related outcome measures and what is routinely measured in surgeons’ outcomes, their findings are nonetheless compelling and call for this elucidation.

Conversely, while Google, Yahoo, social media sites, ReliefWeb and so on can supply situational awareness to potential volunteers as well as host nation communities, enabling them to organise, coordinate and identify sources of support and assistance, they are not peer-reviewed and may not be factual. Noelle Sullivan has written for Scientific American 36 and Huffington Post 37 regarding ‘medical voluntourism’, reporting examples of untrained volunteers performing surgical procedures. In some of her examples, she makes no mention of whether or not a qualified surgeon was overseeing the procedure (as is the practice in virtually every surgical residency in the USA.) In fact, one of the ‘voluntourists’ she named has written a rebuttal with that very piece of information—that she was assisting trained professionals.38 Thus, cross-disciplinary reporting can sensationalise and undermine good-faith efforts that may actually be well designed and effective.

Finally, the economics of surgical and medical missions impact outcomes reporting. The drive to keep overhead costs low results in a vicious cycle39 that ‘is leaving nonprofits so hungry for decent infrastructure that they can barely function as organizations, let alone serve their beneficiaries. The cycle starts with funders’ unrealistic expectations about how much running a nonprofit costs and results in nonprofits’ misrepresenting their costs while skimping on vital systems – acts that feed funders’ skewed beliefs’.40 Although we cannot fully address economic impact in this article, it is a critical obstacle and others have examined its role.41

Suggested mechanisms for reporting outcomes