GRADE Series - Sharon Straus, Rachel Churchill and Sasha Shepperd, Guest Editors
GRADE guidelines: 9. Rating up the quality of evidence

https://doi.org/10.1016/j.jclinepi.2011.06.004Get rights and content

Abstract

The most common reason for rating up the quality of evidence is a large effect. GRADE suggests considering rating up quality of evidence one level when methodologically rigorous observational studies show at least a two-fold reduction or increase in risk, and rating up two levels for at least a five-fold reduction or increase in risk. Systematic review authors and guideline developers may also consider rating up quality of evidence when a dose–response gradient is present, and when all plausible confounders or biases would decrease an apparent treatment effect, or would create a spurious effect when results suggest no effect. Other considerations include the rapidity of the response, the underlying trajectory of the condition, and indirect evidence.

Introduction

Key points

  • GRADE includes three criteria for rating up quality of evidence particularly applicable to observational studies.

  • Rating up one or even two levels is possible when effects in observational studies are sufficiently large, particularly if they occur over short periods of time.

  • A dose–response gradient, or a conclusion that plausible residual confounding would further support inferences regarding treatment effect, may also raise the quality of the evidence.

In prior papers in this series devoted to exploring GRADE's approach to rating the quality of evidence and grading strength of recommendations, we have dealt with issues of framing the question; introduced GRADE's conceptual approach to rating the quality of a body of evidence; and presented five reasons for rating down the quality of evidence: risk of bias, imprecision, inconsistency, indirectness, and publication bias. This ninth article in the series examines the criteria for rating up the quality of evidence.

The three primary reasons for rating up the quality of evidence are (Table 1) as follows:

  • 1.

    When a large magnitude of effect exists,

  • 2.

    When there is a dose–response gradient, and

  • 3.

    When all plausible confounders or other biases increase our confidence in the estimated effect.

We have noted previously that GRADE is relevant to rating evidence regarding the impact of interventions on patient-important outcomes—rather than, for instance, prognostic studies that identify patient characteristics associated with desirable or adverse outcomes. Using the GRADE framework, evidence from observational studies is generally classified as low. Unsystematic clinical observations are usually at a high risk of bias and therefore generally receive a rating of very low quality evidence. There are times, however, when we have high confidence in the estimate of effect from such studies. GRADE has therefore suggested mechanisms for rating up the quality of evidence from observational studies.

The circumstances under which we may wish to rate up the quality of evidence for intervention studies will likely occur infrequently and are primarily relevant to observational studies (including cohort, case–control, before–after, and time series studies) and to nonrandomized experimental or interventional studies (e.g., providing treatment to one of the two matched groups). Indeed, although it is theoretically possible to rate up results from randomized control trials (RCTs), we have yet to find a compelling example of such an instance.

Section snippets

Large magnitude of effect

For some clinical interventions (e.g., hip replacement to reduce pain and functional limitations in severe osteoarthritis, epinephrine to prevent mortality in anaphylaxis, and insulin to prevent mortality in diabetic ketoacidosis), clinicians are, correctly, extremely confident of their effectiveness. Moreover, in each of these situations we are also extremely confident that the impact of the intervention is substantial. Thus, using GRADE's definition of quality of evidence, the underlying

Dose–response gradient

The presence of a dose–response gradient has long been recognized as an important criterion for believing a putative cause–effect relationship [16]. Such a gradient may increase our confidence in the findings of observational studies and thus enhance the assigned quality of evidence (Table 1).

For example, our confidence in the results of observational studies that show an increased risk of bleeding in patients who have supra-therapeutic anticoagulation levels is increased by the finding that

Plausible confounding can increase confidence in estimated effects

Rigorous observational studies will accurately measure prognostic factors associated with the outcome of interest and will conduct an adjusted analysis that accounts for differences in the distribution of these factors between intervention and control groups. The reason that in most instances we consider observational studies as providing only low-quality evidence is that unmeasured or unknown determinants of outcome unaccounted for in the adjusted analysis are likely to be distributed

Other considerations

Particular design features of extremely rigorous well-conducted observational studies may warrant consideration for rating up quality of evidence. For instance, a case-control study found that sigmoidoscopy was associated with a reduction in colon cancer mortality for lesions in range of the sigmoidoscope (OR 0.30, 95% CI: 0.19, 0.48), but not beyond the range of the sigmoidoscope (OR 0.96, 95% CI: 0.61, 1.50) [28]. Possible bias because of unmeasured confounders should have been very similar

A final note of caution

Consideration of all our previously presented criteria for rating down quality of evidence (risk of bias, imprecision, inconsistency, indirectness, and publication bias) must precede consideration of reasons for rating up quality. The decision to rate up should only rarely be made if serious limitations are present in any of these areas. In particular, decisions to rate up because of large or very large effects should consider not only the point estimate but also the width of the CI around that

Conclusions

In summary, there are three factors that might increase the quality of evidence. In general, these three factors, mostly applicable to observational studies, are encountered infrequently. Although most observational studies, even if well done, yield low-quality evidence, one can consider rating up the quality of evidence when there is a large or a very large magnitude of effect, when consideration of all plausible residual confounders and biases would reduce a demonstrated effect, or suggest a

References (28)

  • S.J. Weiner

    Contextualizing medical decisions to individualize care: lessons from the qualitative sciences

    J Gen Intern Med

    (2004)
  • Rothman K, Greenland S. Modern epidemiology. 2nd ed. Lippincott Raven;...
  • P. Glasziou et al.

    When are randomised trials unnecessary? Picking signal from noise

    BMJ

    (2007)
  • S.C. Cannegieter et al.

    Thromboembolic and bleeding complications in patients with mechanical heart valve prostheses

    Circulation

    (1994)
  • Cited by (0)

    1

    The GRADE system has been developed by the GRADE Working Group. The named authors drafted and revised this article. A complete list of contributors to this series can be found on the journal's Web site at www.elsevier.com.

    View full text