Registration of observational studies

Observational studies, such as cohort and case-control studies, are an important form of medical research, but they are also vulnerable to bias and selective reporting.1 They often produce large datasets that can be subjected to multiple analyses. Researchers may then craft a paper that selectively emphasises certain results, often those that are statistically significant or provocative. These decisions may reflect strong financial or academic interests and prior beliefs. At present, consumers of observational research cannot easily distinguish hypothesis driven studies from exploratory, post hoc data analyses. Researchers do not routinely disclose the number of additional analyses performed. Nor is there any satisfactory way to know whether the research questions or methods of statistical analysis diverged from those initially planned. It has been observed that there is “little or no penalty” for data dredging and selective reporting. Rather than attracting censure it can “get you into the BMJ and the Friday papers.”2

In the linked article (doi: 10.1136/bmj.b4184), Hemingway and colleagues reinforce many of these arguments, particularly with respect to studies of prognosis, because these can be important clinically but are often flawed.3 This group, which includes two of the BMJ’s statistics editors, Doug Altman and Richard Riley, recommends that “all research on humans should have a protocol.” Such calls for registries of observational research are gathering pace, and indeed an international meeting held in London last September was devoted entirely to the discussion of such registries and other efforts to improve the credibility of observational research.4 5

The BMJ publishes a large amount of observational research and has an important stake in its quality. We are now actively supporting the registration of observational study protocols and results in publicly accessible registries. Although the BMJ does not advocate one particular registry, we note that around 14 000 observational studies are already registered with clinicaltrials.gov, and that the results of such studies can be posted there too, as is already the case for clinical trials.6 The development of registries for randomised trials was driven by several ethical and scientific concerns, not just by the desire to prevent suppression of unfavourable results (box).7 We feel strongly that most of these points also apply to observational studies.

We recognise the lack of consensus on this proposal. In a linked editorial (doi:10.1136/bmj.c703), Sørensen and Rothman express concerns that the insistence of journals on protocols and registration would be too restrictive, and they argue that peer reviewers and editors are as much to blame as researchers for the publication of low quality work.8 There are legitimate worries, too, that prioritising protocol driven studies might discourage publication of genuinely important results that emerge from data mining or that it might have other unintended negative effects because “subgroups and multiple analyses are a necessary part of observational research: otherwise, one cannot make new discoveries, nor quickly check discoveries by others.”9 We agree that exploratory observational research is important. Many new ideas arise from unexpected findings in observational research, and many researchers learn their skills from examining available datasets. However, that is not the sort of research the BMJ usually aims to publish; rather, we give highest priority to studies that provide strong support for inferences applicable to clinical practice. We think the case against data driven observational studies is particularly compelling under these circumstances.

We understand concerns that extending these rules to observational studies might encourage editors—particularly of general journals—to be overzealous or clumsy in their application. The STROBE statement has improved reporting of observational studies by asking authors to spell out in their papers exactly what they did during their studies.10 It asks authors to “explain the scientific background and rationale for the investigation being reported” and “state specific objectives, including any prespecified hypotheses.” As journal editors, we have probably not paid enough attention to emphasising these points, but we aim to do so from now on. However, like most reporting statements, STROBE is aimed at improving the clarity of study reporting and comes too late to influence study design.

For these reasons, we will now ask authors of papers reporting observational studies submitted to the BMJ to tell us more about the origins, motivations, and data interrogation methods of that work. This may not be appropriate for all observational studies, and we aim to apply the policy in a flexible and thoughtful manner. We would not reject an observational study just because it did not have a prespecified hypothesis, but we would want the exploratory nature of its research question, and its design, to be fully reported.

Among other things, we will be asking authors to report in their papers a clear statement of whether the study hypothesis arose before or after inspection of the data (and, if afterwards, we will need an explanation of steps taken to minimise bias); we will ask to see study protocols if they exist; and we will add to the papers’ abstracts their registration details, if they have been registered. If the study is registered we will ask whether the protocol was registered before data acquisition or analysis began.

Registration of observational studies is just one of many changes needed to increase confidence in observational research, but we believe it is the crucial next step. The aim is to facilitate the design and reporting of observational research, not to hinder it. Trial registration has had a substantial and important positive effect on the design, conduct, and reporting of randomised clinical trials, and we believe it is time to extend those benefits to observational research.