Article Text
PDF
Abstract
Background Research integrity and research fairness have gained considerable momentum in the past decade and have direct implications for global health epidemiology. Research integrity and research fairness principles should be equally nurtured to produce high-quality impactful research—but bridging the two can lead to practical and ethical dilemmas. In order to provide practical guidance to researchers and epidemiologist, we set out to develop good epidemiological practice guidelines specifically for global health epidemiology, targeted at stakeholders involved in the commissioning, conduct, appraisal and publication of global health research.
Methods We developed preliminary guidelines based on targeted online searches on existing best practices for epidemiological studies and sought to align these with key elements of global health research and research fairness. We validated these guidelines through a Delphi consultation study, to reach a consensus among a wide representation of stakeholders.
Results A total of 45 experts provided input on the first round of e-Delphi consultation and 40 in the second. Respondents covered a range of organisations (including for example academia, ministries, NGOs, research funders, technical agencies) involved in epidemiological studies from countries around the world (Europe: 19; Africa: 10; North America: 7; Asia: 5; South-America: 3 Australia: 1). A selection of eight experts were invited for a face-to-face meeting. The final guidelines consist of a set of 6 standards and 42 accompanying criteria including study preparation, protocol development, data collection, data management, data analysis, dissemination and communication.
Conclusion While guidelines will not by themselves guard global health from questionable and unfair research practices, they are certainly part of a concerted effort to ensure not only mutual accountability between individual researchers, their institutions and their funders but most importantly their joint accountability towards the communities they study and society at large.
- epidemiology
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Key questions
What is already known?
Research integrity and research fairness have gained considerable momentum in the past decade and have direct implications for global health epidemiology.
Balancing research integrity with the realities of conducting fair global health epidemiological research can be challenging.
Unfortunately, existing good epidemiological practice guidelines developed by national epidemiological associations are not tailored to the idiosyncrasies of global health and lack international legitimacy.
Also, existing guidelines for research fairness are not specific to epidemiology.
What are the new findings?
Through a Delphi consultation study involving a wide range of experts with experience and expertise in global health research and epidemiology, we developed guidelines for good epidemiological practice in global health that address the core principles of research integrity and fair global health research.
The final guidelines consist of 6 standards and 42 accompanying criteria including study preparation, study protocol and ethical review, data collection, data management, analysis, reporting and dissemination.
Key questions
What do the new findings imply?
Guidelines will not by themselves guard global health from questionable and unfair research practices.
However, guidelines are certainly part of a concerted effort to ensure the accountability of funders, institutions and researchers towards the communities they study and society at large.
We invite all stakeholders involved in the commissioning, conduct, appraisal and publication of global health research to consider the use of these guidelines in their research.
Introduction
Bora kujenga daraja kuliko ukuta
Better build bridges than walls (Kiswhahili proverb)
Global health epidemiology studies the causes and consequences of morbidity and mortality across geographical boundaries, with emphasis on equitable disease control and health promotion in low-income and middle-income countries. Research integrity and research fairness have gained considerable momentum in the past decade and have direct implications for global health epidemiology. In this article, we argue that research integrity and research fairness principles should be equally nurtured by global health epidemiologists who aim to produce high-quality impactful research—but bridging the two can lead to practical and ethical dilemmas. In the light of these reflections, we propose guidelines for epidemiological studies targeted at stakeholders involved in the commissioning, conduct, appraisal and publication of global health research.
Research integrity has emerged as a response to the ‘reproducibility crisis’ (the inability to reproduce research findings), which has shaken the foundations of most scientific disciplines.1 Within epidemiological research findings obtained from ill-designed, badly implemented, inappropriately analysed or selectively reported studies will also lead to irreproducible results.2–4 Data fabrication, falsification and plagiarism represent the most extreme case of scientific misconduct and consequently inability to reproduce research findings. Yet, practices in the grey zone between this type of deliberate misconduct and ideal scientific behaviour—denoted as ‘questionable research practices’5 or ‘research waste’6—may be more prevalent and ultimately, more damaging.7 The reproducibility crisis can be partly attributed to the limits of hypothesis testing paradigms,3 8 9 but lack of scientific rigour is also a major contributor.7 10 11 A number of scientific regulatory bodies have issued documents over the past decade to foster research integrity and thereby tackle questionable research practices and research waste, either in the form of codes of conduct for researchers (such as in the European Union (EU)12 and in India13) or guidelines and policies (eg, Tanzania14 and Uganda15).
Definitions of research integrity centre around four principles: reliability, honest, respect and accountability.12 Efforts to foster research integrity in the EU have resulted in a push for open science in all scientific areas.16 In epidemiology, more specifically a number of guidelines for good epidemiological practice have also been developed, such as in Switzerland,17 Germany,18 the Netherlands19 and France.20 Yet, as we argue in following paragraphs, global health epidemiologists also need to reckon with the idiosyncrasies of conducting global health research,21 which derive from its multidisciplinary nature and emphasis on transnational issues and equity at population level.22
Multidisciplinary methods are essential in global health research as most issues’ studies are embedded in complex systems centred around human behaviour. The study of complex systems in turn is closely related to reproducibility: even when methods have been implemented properly, contextual factors have a key role and can thereby affect reproducibility of findings. The biomedical focus on finding ‘what works’ means that experimental designs are considered the gold standard to evaluate public health interventions,23 and reproducibility is often considered purely in terms of statistical uncertainty. But as critics of counterfactual methodologies in development have argued, these methods produce very context-specific evidence (‘did it work there and then’), which risks overlooking specific sociopolitical and cultural contexts.23 24 Social sciences (eg, anthropology, sociology, political economy) are uniquely placed to understand the social conditions related to the success (or failure) of interventions and provide essential information about reproducibility to complement statistical uncertainty. This criticism can be extended to the use of experimental designs in global health and means that in order to produce useful evidence to decision-makers25 global health epidemiologists need to engage with other related disciplines—and most notably social sciences—in multidisciplinary, interdisciplinary or transdisciplinary research.26
Global health’s emphasis on transnational issues and equity, on the contrary, implies that research integrity needs to be expanded to include research fairness principles. One of the principles of research integrity is respect—defined as ‘respect for colleagues, research participants, society, ecosystems, cultural heritage and the environment’.12 Strictly speaking, the transnational nature of global health refers to the study of determinants and solutions that cross national boundaries, such as climate change or urbanisation.22 But in practice—for a host of historical reasons27—transnational research often implies transnational research collaborations and more specifically partnerships between institutions in higher and lower income countries. The power imbalances potentially arising in such partnerships are at the centre of research fairness concerns28–32 and are broadly aligned with calls to decolonise global health.33–36 More specifically, research fairness aims at redressing some of the power imbalances in global health, which prevent local stakeholders from shaping the research agenda and competing on a level playing field in scientific arenas. In doing so, research fairness seeks to maximise the positive impact of global health research both on local researchers and on local communities. As such, research fairness lies at the core of the principle of respect in global health partnerships.
A broad understanding of research integrity that encompasses research fairness is therefore key to high-quality and impactful global health research—yet there is evidence of shortcomings on both fronts currently. While research integrity in global health has not been amply studied, questionable research practices appear to be equally widespread in this field as in any other.21 Many different factors can lead to misconduct in research, and global health is not immune to any of these.37 38 Also, two recent studies have shown that research fairness—or lack thereof—plays a central role in global health collaborations between researchers in both high-income and low-income countries39 and can lead to high levels of moral distress.40 Reported challenges included failure to recognise the scientific merit of local staff, the absence of local benefits for the populations studied,39 inequitable allocation of public health resources, conflicts of interest, lack of autonomy among community members or beneficiaries of programmes and inadequate access to essential services.40
In our experience, balancing research integrity with the realities of conducting fair global health epidemiological research can be challenging. Unfortunately, existing good epidemiological practice guidelines developed by national epidemiological associations17–20 are not tailored to the idiosyncrasies of global health and lack international legitimacy. Also, existing guidelines for research fairness are not specific to epidemiology.28 30 41 42 In table 1, we describe potential tensions that can arise in each of the study phase when trying to align the principles of research integrity and research fairness within the frame of global health practice, by referring to the four main tenets of global health research described above: multidisciplinary, transnational, equity focused and at population-level.
Potential tensions reconciling research integrity and research fairness principles in global health epidemiological practice
Against this backdrop, we embarked on a study to develop guidelines for good epidemiological practice in global health targeted at stakeholders involved in the commissioning, conduct, appraisal and publication of global health research. The aim of this study was to develop guidelines in which the principles of research integrity are applied while at the same time acknowledging the needs and realities of conducting fair global health research.
Methods
Preliminary guidelines
Between October 2017 and March 2018, we conducted a literature review and developed a preliminary set of guidelines for internal use at KIT Royal Tropical Institute, following the AGREE II methodology.43 The development consisted of four steps (figure 1) .
Overview of guideline development steps. GEP, good epidemiological practice.
First, we performed online searches in Google and PubMed to gather information on existing best practices for epidemiological studies by combining the word ‘epidemiology’ or ‘epidemiological’ with the following keywords: ‘guidelines’, ‘guidance’, ‘procedures’, ‘standards’, ‘practice’, without any time period restrictions. Results from this search were critically appraised for relevance. In addition, we selected relevant reporting guidelines from the EQUATOR network website.44 The records identified through this search could be classified into the following categories: (1) ethics guidelines for research45–48; (2) guidelines for good epidemiological practice laid out by international49 and national epidemiological associations17–20 and (3) academic reporting guidelines.50–54
Second, we identified key elements of research fairness32 42 55 and global health22 that have an impact on epidemiological practice. These have been described in more detail elsewhere21 and consist of (1) stakeholder involvement; (2) fair and equal partnerships; (3) multiple ethical reviews; (4) emergency research; (5) equity, gender and intersectionality; (6) multidisciplinary research; (7) secondary analyses of existing data; (8) fair data sharing.
Third, we synthesised existing good epidemiological practice guidelines from our literature search and adapted them in order to ensure an alignment with key elements of global health and research fairness. The output of this exercise was a set of standards and criteria. We opted for the following widely used (though not unanimously agreed on56) convention: standards are qualitative descriptions of the overall quality expectation for the given step in the process; each standard has a number of corresponding criteria, which constitute a measurable checklist to assess whether the standard has been met.
Lastly, these guidelines were internally validated at KIT by means of three consultative sessions. The first session focused on study preparation, protocol development and ethical review; the second on quality assurance, data collection and data management; and the third on data analysis, reporting dissemination and data storage. A multidisciplinary group of KIT global health researchers and advisors participated in these sessions, including four epidemiologists, five public health doctors and two social scientists. Two external expert reviewers (an epidemiologist from Nigeria and one from Bangladesh) also provided feedback on these guidelines.
Validated guidelines: Delphi study
We sought external validation of the KIT guidelines by means of a Delphi study, as recommended in the literature on guideline development.57 58 The Delphi method is a structured iterative process to incorporate the views of a group of experts on a draft product in order to develop a final version that represents agreement by all participants. For the development of guidelines, the Delphi method consists of repeated surveys with participants to assess the level of agreement with each element (standards and criteria) of the guidelines and to collect suggestions for change. The respondents’ comments are used to formulate a refined version of the guidelines, which is then submitted for a next round of feedback until a satisfactory level of agreement is reached.58–60
The Delphi study was designed in three stages (figure 1): two online ‘e-Delphi’ consultations61 with a larger group of experts and one face-to-face meeting with a selected group of respondents. We conducted a stakeholder analysis62 to select a diverse group of experts representing the major organisations involved in the commissioning, conduct, appraisal and publication of global health research. As part of this analysis, we considered each stakeholder’s interest and position towards good epidemiological practice and their influence on the implementation of studies within and beyond their organisation. We identified and contacted individuals working or affiliated with these organisations through KIT’s professional network. To reflect the multidisciplinary and transnational nature of global health research, we aimed to identify experts from a wide range of disciplines and countries.
The tools for online consultation consisted of online SurveyMonkey questionnaires. Before sending tools to study participants, we pilot-tested the tools by requesting KIT epidemiologists who were not involved in this study to complete the survey.
The first round of online consultations was conducted in October and November 2018. This round consisted of open-ended questions to encourage qualitative input from experts. Participants were presented with the guidelines through an online questionnaire and were asked to indicate whether each standard and criterion should be kept, deleted or revised. There was also room for additional comments per standard/criterion as well as comments pertaining to the guidelines in general.
The second round of consultations was conducted in November and December 2018. For transparency and clarity, participants received a document describing the old and revised guidelines including comments on the changes and reformulations that were made. This round was more quantitative in nature. Per standard and criterion, participants were asked to rate whether they agreed with the revised formulations using a 5-point Likert scale (strongly agree to strongly disagree), with opportunities for general comments. The data from this round was analysed by calculating the percentage agreement for each standard and criterion from the Likert scales. Following guidance from Hasson et al,59 we did not set an a priori cut-off value for consensus, as there is no universally agreed value.
Finally, a face-to-face meeting was organised in Amsterdam in June 2019 with a selection of experts who had participated in the online consultations. The aim of this meeting was to take stock of the online consultation responses, to resolve issues that remained thorny despite two rounds of consultation and to agree on the final guidelines. Consensus was reached through iterative rounds of group discussions leading to proposals which were tested in plenary discussions. Where needed, proposal amendments were discussed and further tested for agreement until a satisfactory level of acceptance from all participants was reached.
Ethical review
Prior to conducting the Delphi consultation, the KIT Royal Tropical Institute’s Research Ethics Committee was approached to review the study’s tools and protocols. The study was exempted from full ethical review on the following grounds: (1) data collection is by means of voluntary survey, which does not contain any personal questions and only covers information related to respondents’ duties; (2) since it is an online survey, participants can decline or withdraw participation at any moment without this having any consequences. While we asked participants for their names and affiliations, this was not compulsory.
Participants and public involvement
Participants received a draft version of this article prior to its submission and the final article will be shared on publication. We considered this an appropriate means of dissemination since all participants are closely involved in academic research. In addition, the website bridge-statement.org will include additional dissemination material, such as translations of this article (in French, Spanish) and training material for epidemiology and global health postgraduate courses.
Results
Participants
We invited 163 people by email to contribute to the first round of the Delphi validation exercise. Forty-five (28%) participants agreed to participate and completed the first round. Participants in the first round represented various disciplines and organisations (figure 2) and came from a wide range of countries (figure 3). Forty experts participated in the second round. Despite the drop in participants, we were able to keep the same balance of geographical and disciplinary representation in both rounds. Finally, for the face-to-face meeting in Amsterdam, we selected eight experts who had participated in the online consultations. These were chosen based on their professional and geographical background, as well as their input in the previous rounds. We therefore invited four delegates from the Netherlands and Belgium, and four international delegates from India, Tanzania, Uganda and Peru.
Number of e-Delphi round 1 respondents by type of organisation (N=45) [1]. [1]One individual can belong to more than one type of organisation.
Number of e-Delphi Round 1 respondents by region (N=45) [1]. [1]Refers to individuals’ working location, which is at times different from their country of origin.
Revision of guidelines
The preliminary guidelines consisted of 51 criteria across nine standards covering the following nine implementation steps for the conduct of an epidemiological study : (1) study preparation; (2) study protocol and ethical review; (3) quality assurance; (4) statistical analysis plan; (5) data collection; (6) data management; (7) analysis; (8) reporting and dissemination; (9) data storage.
Overall, the e-Delphi consultation led to a substantial simplification and shortening of the guidelines. After the first round of online consultations, 14 criteria out of the original 51 were deleted (27%), 5 were added, and all others were reworded. The statistical analysis and quality assurance standards were removed as they were considered too detailed and prescriptive. In their comments, participants warned against the risk of having too many standards and criteria as well as overlap with other existing guidelines. Furthermore, the data storage standard was dismantled and criteria were placed in the data management and dissemination standards. We revised the criteria that were considered important but not always attainable or applicable, with wording such as ‘strive for’ and ‘consider what is in the remit’ to ensure they were reflected but not considered mandatory.
A total of 6 standards and 45 criteria were taken into the second round of online consultations. Overall, there was a high level of agreement in the second round, with all but two criteria achieving at least 85% of the 40 respondents grading as ‘strongly agree’ or ‘agree’ that the standards/criteria should be part of the final guidelines in their revised formulation. The criterion with the highest disagreement was the data analysis criteria ‘Analysts should not work with data that have direct personal identifiers’ (undecided: 24%; disagree: 8%; strongly disagree: 3%). The main dissatisfaction was that direct identifiers might be date of birth or a postcode and may be needed for analyses. Instead, respondents favoured a focus on a limited number of people handing personal data, alongside training for data handlers on confidentiality and data security. The criterion with the second highest disagreement was the protocol development criterion: ‘The protocol should include a clear and complete data statistical analysis plan which excludes data-driven adaptations’ (undecided: 11%; disagree: 8%). This criterion was considered unclear, rigid, unrealistic and potentially leading to inadequate analysis strategies. Instead, respondents favoured a stronger focus on transparency in the reporting stage.
During the face-to-face meeting in Amsterdam, we further streamlined and simplified the standards and criteria taking into account the qualitative feedback from all survey respondents. The outcome of the overall study consists of a final set of 6 standards and 42 accompanying criteria covering the following steps in the study process: (1) study preparation; (2) study protocol and ethical review; (3) data collection; (4) data management; (5) analysis; (6) dissemination and communication (table 2). A glossary of terms can be found in an accompanying explanation and elaboration paper (REF).63
Final BRIDGE checklist
Discussion
Based on literature and expert opinion, we have compiled a set of standards for good epidemiological practice in global health, bridging research integrity and research fairness. While there is an inherent tension between efforts to conduct research according to strict technical guidelines of good epidemiological practice and the realities of conducting global health research, we believe these guidelines offer a practical support to epidemiologists to navigate through the complex global health landscape.
Our aim was not to duplicate existing work, but rather to bring together existing principles in one overarching guideline with a focus on practical implications for researchers. While this article has focused on the justification and methodology followed for the development of the guidelines, further explanation and elaboration on all standards and criteria can be found in the accompanying publication (REF).63 Wherever possible in our explanation and elaboration document, we have made reference to existing tools and guidelines.
There are a number of limitations to our study with a potential impact on the resulting guidelines. First, these guidelines are based on a literature review that was conducted in 2017. While we made all attempts to supplement this review with relevant literature that we encountered thereafter, it is possible that we omitted important recent developments in the field of global health epidemiology and research fairness. Second, our recommendations are very much influenced by the opinions and experience of participants of the Delphi consensus study, and more specifically the experts which participated in the face-to-face meeting. While we made all attempts to ensure a diverse group of expertise and geographical background reflecting the target group for these guidelines, it is possible that some perspectives may not have been given duly prioritised. Third, through the e-Delphi validation process, the original guidelines lost a number of features which made them very specific to epidemiology (eg, an entire standard on steps involved in the development of statistical analysis plan was dropped). As a result, it could be argued that the current guidelines reflect good research practice more broadly, rather than good epidemiological practice specifically. The accompanying explanation and elaboration notes (REF) do provide further epidemiological details for a number of item of the guidelines.
Nevertheless, we hope that the greatest benefit of these guidelines will be to contribute to current efforts in global health seeking to place communities, societies and researchers from low-income and middle-income countries at the heart of research endeavours in their own countries. With regard to researchers more specifically, we hope these guidelines can improve the quality of epidemiological research by offering explicit recommendations for epidemiologists who are uncertain about how to proceed and challenge the beliefs of researchers accustomed to outmoded practices.64 These guidelines may also be considered to support quality improvement activities, by providing the bases for the design of quality assessment (eg, audits) and quality assurance (eg, risk analysis) tools.64
We are aware that guideline implementation is a much more challenging process than guidelines development. One of the major hurdles is that guidelines can be seen as inconvenient and time-consuming by practitioners.64 In global health research, challenges around implementation are compounded by the great heterogeneity in study contexts and large number of involved parties,37 38 resulting in a blurring of responsibilities and conflicting incentives and priorities.65 Furthermore, epidemiological practice itself varies widely in scope. The sophistication of strategies to ensure their successful completion should be commensurate to their level of complexity. Factors influencing complexity include the type of setting, the number of study participants, the number of study sites, the types and quantity of data collected per participant, the duration of the study, the mix of disciplines, and so on. In this regard, we believe the adaptation of these guideline to suit local contexts and specific institutional practices and process will be key for implementation. Guideline adaptation frameworks provide a systematic way of approaching adaptation, and their use may increase transparency, methodological rigour and the quality of the adapted guideline.66
Guidelines will not by themselves guard global health from questionable and unfair research practices. But they are certainly part of a concerted effort to ensure the accountability of funders, institutions and researchers towards the communities they study and society at large. The intent of these guidelines is neither to cripple researchers’ freedom nor to impose inflexible rules on the conduct of studies—but rather to streamline efforts and stimulate humility and reflection in order to generate high-quality and impactful research. We invite all stakeholders involved in the commissioning, conduct, appraisal and publication of global health research to consider the use of these guidelines in their research. We welcome any feedback at gep@kit.nl. All comments received will be considered as part of our planned quinquennial revision.
Acknowledgments
This work was made possible thanks to generosity of many people who selflessly gave their time and shared their insights and knowledge throughout all steps of this study. Heartfelt thanks to Christian Schindler, Stella van Beers and Paul Klatser, Fernando Maldonado, Mirjam Bakker, Lisanne Gerstel and Lindy van Vliet for providing the space for initial reflections on these themes including in the post-graduate training programs of both Swiss TPH and KIT Royal Tropical Institute. We are also very grateful to Gerard Swaen for encouraging us to embark on an external validation study and to Prisca Zwanikken for coaching us on the necessary steps. Special thanks to Christina Mergenthaler who played a key role in drafting the first internal KIT guidelines and many thanks to KIT colleagues who provided critical and thoughtful feedback in the internal consultations sessions: Ente Rood, Lucie Blok, Maaike Flinkenflögel, Barend Gerretsen, Egbert Sondorp, Pam Baatsen, Irene de Vries, Tasneem Kakal, Ingrid Zuleta, Ankie van den Broek, Margo van Gurp, Frouwke Veenstra, Eelco Jacobs. We are indebted to Hassan Abiodun and Sayera Banu for their critical expert external review of the internal KIT guidelines. Thanks to Ewen Le Borgne for great suggestions on how to conduct the face-to-face consensus meeting in Amsterdam. Last but not least, our deep appreciation goes to all the e-Delphi participants for their time and thoughtful input: Seye Abimbola, Brenda Kateera, Frode Forland, Solomon Narh-Bana, Carmen Varela Santos, Lex Bouter, Jennita Reefhuis, Suzanne Verver, Antonio Barnabe Ortiz, Sandro Colombo, Cudjoe Bennet, Maya Subelj, Tracy Glass, Susan van den Hof, Borna Mueller, Michiel de Boer, Miguel Antonio Trelles Centurion.
References
Footnotes
Handling editor Seye Abimbola
Contributors SA conceived the study, led its execution and compiled the first draft of this manuscript. IT coordinated the implementation of the online Delphi consultation study and wrote the methodology description. MS participated in the consultation sessions for the development of the initial guidelines and DJ and MS led the face-to-face expert review meeting. KV, SFR, AL, SJ, DW, HM-K and WM participated in the expert face-to-face meeting and took final decisions with regard to the formulation of the standards and criteria. All authors read and approved the final version of this manuscript.
Funding This study was made possible thanks to funding from the KIT Knowledge Investment Fund
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available in a public, open access repository. 10.5281/zenodo.3903146.