• Epidemiology

Being an epidemiologist means always having to say you're sorry. Epidemiology has constantly been criticised for the problems of lack of randomisation of exposures, misclassification of exposures and outcomes, and inadequate control of confounding.1 Nevertheless, epidemiologists continue to make important scientific discoveries, ranging from Snow's work on cholera2 to the discovery of tobacco smoking3 and asbestos4 as major causes of lung cancer, through to more recent discoveries such as the link between human papillomavirus (HPV) and cervical cancer.5 All of these discoveries took decades to become accepted, but eventually successful public health interventions followed. We must be doing something right.

Of course, we can also get things wrong, and there are many examples of findings which hit the headlines but were not replicated when further studies were carried out.6 It is therefore not surprising that the public may be sceptical of epidemiological findings.6 Perhaps the problem is not that epidemiology is more prone to error than other sciences (basic researchers, for example geneticists, probably get things wrong at least as often, and often hype up their findings even more than we do), but that our mistakes may be more likely to be highlighted by the media. Examples include studies of β carotene and cardiovascular disease, hormone replacement therapy, vitamin E and vitamin C intake in relation to cardiovascular disease, and fibre intake in relation to colon cancer.7

However, these examples of ‘epidemiological failures’ primarily involve studies of lifestyle factors (particularly diet). These are notoriously difficult to investigate, since the exposed group (eg, those with high β carotene levels in their diets) will often be markedly different from the non-exposed group as regards other lifestyle factors. Ironically, some areas of epidemiological research are less prone to error, but these are often areas in which there is most controversy.8 In particular, there are usually only relatively minor problems of confounding in occupational epidemiology, since there are usually only minor differences in smoking, diet, etc, between different groups of workers,9 but this is perhaps the field of epidemiology where the findings are most likely to be disputed.

These controversies are not occurring in a vacuum.8 The difficulties of conducting epidemiological research are exacerbated by the activities of companies unhappy with the findings of such studies. The usual approach is for the company concerned to hire consultants to criticise the research publicly, either when it appears in print or even prior to publication.10 In recent years, these efforts have been further developed and refined with the use of websites and publicity that stigmatises unwelcome research findings as ‘junk science’.11 In some instances these activities have gone as far as efforts to block publication.12 Recent examples include attempts to influence studies on the toxicity of benzene13 and diesel particulate matter,14 the various industry efforts over many years to influence the conduct and interpretation of research into the health effects of dioxin,15 the industry campaign to undermine an OSHA chromium (VI) standard16 and corporate infiltration of a panel convened to set standards for chromium (VI) in California.17 More recently, epidemiology in general, and occupational epidemiology in particular, have been criticised for a inherent tendency to produce false positive findings,18 a view which has been disputed by other epidemiologists including myself.19

These controversies set the context for current proposals to require the registration of observational studies. These were discussed at a workshop20 organised by the European Centre for Ecotoxicology and Toxicology of Chemicals (ECETOC), and are summarised and supported by the paper by Rushton (see page 84).21 The workshop report argued that registration of clinical trials is now required by law, or by ethics committees, in many parts of the world, and that this should be extended to epidemiological studies. This proposal has received largely critical comments from epidemiologists22–28 but some support from medical journals.29 30 I was invited to participate in the workshop, but declined, since I considered it to be inappropriate for such a workshop to be sponsored by an agency funded by the chemical industry.29 Furthermore, the task of fostering good epidemiological practice is already being undertaken by professional societies such as the International Epidemiological Association,31 and through the STROBE guidelines.32

There are several reasons why the proposal to require registration of epidemiological studies is unnecessary and misguided, and would do more harm than good, both in scientific and in public health terms. These stem from the differences between randomised clinical trials and observational studies, which were not recognised by the ECETOC Workshop report.22 27 28

First, the reasons for the requirement for the registration of clinical trials are not the same as those that have been proposed for observational studies. The main reason for registration of clinical trials is to avoid the non-publication of true negatives (the ‘file-drawer’ problem24) where an intervention is found not to work, or even to be harmful, and the investigators, or the study funders, are reluctant to publish the findings (a recent example of this is the Vioxx scandal).33 34 In contrast, the main reason proposed for the registration of observational studies is that they may produce false positive findings, which may not have been a priori hypotheses, but which may be ‘cherry picked’,24 and may receive more media attention than is warranted. This is a different problem, which requires a different solution. In particular, we should not be making any decisions on the basis of a single study—we should be considering the totality of the evidence. When evaluating a particular study, we should be asking, “What was the evidence for this hypothesis before the study was done, and what information does this study add?” not “What was in the mind of the investigator before the study was started?”. A finding which confirms an a priori hypothesis may be unconvincing if there was little prior evidence for the hypothesis, whereas a finding based on a post hoc hypothesis may be quite convincing if there was strong prior evidence (whether or not this was known to the investigator beforehand). For example, the finding that cat ownership may actually protect against the development of asthma, was apparently first discovered by accident and then confirmed by post hoc analyses of a number of existing data sets.35 Of course, a study may assess a large number of risk factors (eg, occupational cancer case–control studies using occupational titles), many of which do not have any a priori evidence of risk, but this problem can be readily addressed using semi-Bayesian methods.36 Furthermore, the problem of publication bias can be readily addressed by making full tables of findings available online. In addition, it is straightforward for editors to request that “authors report in their papers a clear statement of whether the study hypothesis arose before or after the inspection of the data”,30 which does not require pre-registration of studies.

Second, clinical trials involve interventions, whereas observational studies do not.26 Thus, there is a deeper ethical obligation to the participants to publish information so acquired, both because the participants have assented to a degree of risk in being randomised to what may be an inferior treatment, and because that information may have more direct impact and relevance to intervention decisions. As Vandenbroucke notes,27 “there is a fundamental divide between the purpose of clinical trials, where results may determine medical interventions in millions of patients, and that of observational research that searches to understand the occurrence of disease”. The former approach not only raises major issues about monitoring and reporting of adverse events, but also means that the findings are more directly relevant to health policy and clinical practice. On the other hand, an observational study that reports, for example, that a particular chemical is associated with an increased risk of a particular type of cancer, is very unlikely to produce immediate policy decisions; rather, it is likely to feed into a large body of other evidence, for example in the International Agency for Research on Cancer (IARC) Monograph series.37 Even when intervention is seen as needed, the form of that intervention is left wide open by evidence of risk (eg, do we seek to eliminate or merely reduce the exposure? And how to do either?), whereas by its very nature a trial incorporates an explicit intervention protocol.

Third, “the proposal is at odds with the way that progress is made in medical science”,27 and in science in general. “Historians and philosophers of science would recoil at the notion than advance registration of all scientific studies… would produce better science”.25 Should Darwin have pre-registered his hypotheses before commencing the voyage of the Beagle? And would his theory of natural selection be more valid if he had? Should Watson and Crick have pre-registered their studies of DNA (if they had, they probably would have been stopped because of competing claims from other researchers and interference from the British Medical Research Council).38 Should we pre-register our studies of the health effects of climate change? Observational sciences, including cosmology, evolutionary biology, ecology, geology and many others, do not proceed in the neat and organised way that is the case (to some extent) with randomised controlled trials. Observational science is so much more messy, exciting and valuable, than that.

Fourth, some protocol adaptations may improve, rather than compromise, the validity of the research.22 These may improve recruitment, allow more accurate measurements (eg, by incorporating new laboratory methods), achieve better control of confounding (eg, by controlling for risk factors discovered after the original protocol was written) or incorporate improved methods of data analysis (eg, Bayesian methods for multiple comparisons). There is a danger that editors and reviewers will be encouraged to simply compare submitted manuscripts with registered protocols.22

Finally, the proposal for registration of observational studies adds yet another obstacle to initiating, conducting and publishing epidemiological research, an obstacle which is “unjustified and insidious”.26 I have previously written that “for every epidemiologist trying to change a light bulb, there are now several critics hired by industry to argue that they are doing it the wrong way, or that it is not broken and doesn't need changing at all, or that they have changed it the wrong way and should do it again”.39 We can now also be criticised for not registering our intention to change the light bulb in advance. Inevitably, this proposal will be misused to prevent or delay the publication of inconvenient findings,8 12 39 despite the best of intentions on the part of the workshop participants.

Of course, it is difficult to oppose any measure that appears to be fostering better epidemiological practice. No-one would oppose better control of confounding, so who would oppose better documentation and conduct of epidemiological studies? There is plenty of room for improvement, so why would we oppose any measure that aims to improve the quality of epidemiological research? Of course we should always attempt to improve the quality of our research, but the proposal for registration of epidemiological studies will not achieve this goal. At best it will be a waste of time, and at worst it will be hazardous for science, and for public health. There are plenty of better ways to improve the quality of published epidemiological research, including better promotion and enforcement of rules for good epidemiological practice,31 32 and better training of researchers, reviewers, reporters, and above all, editors.25