The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies

https://doi.org/10.1016/S0167-6296(98)00039-3Get rights and content

Abstract

The literature which considers the statistical properties of cost-effectiveness analysis has focused on estimating the sampling distribution of either an incremental cost-effectiveness ratio or incremental net benefit for classical inference. However, it is argued here that rules of inference are arbitrary and entirely irrelevant to the decisions which clinical and economic evaluations claim to inform. Decisions should be based only on the mean net benefits irrespective of whether differences are statistically significant or fall outside a Bayesian range of equivalence. Failure to make decisions in this way by accepting the arbitrary rules of inference will impose costs which can be measured in terms of resources or health benefits forgone. The distribution of net benefit is only relevant to deciding whether more information is required. A framework for decision making and establishing the value of additional information is presented which is consistent with the decision rules in CEA. This framework can distinguish the simultaneous but conceptually separate steps of deciding which alternatives should be chosen, given existing information, from the question of whether more information should be acquired. It also ensures that the type of information acquired is driven by the objectives of the health care system, is consistent with the budget constraint on service provision and that research is designed efficiently.

Introduction

The recent discussion of the statistical properties of cost-effectiveness analysis has considered the sampling distribution of an incremental cost-effectiveness ratio for classical statistical inference. Here it is argued that the problems associated with defining a reliable test statistic for an incremental cost-effectiveness ratio can be overcome once a price per effectiveness unit has been determined. The decision rule and the test statistic can then be based on incremental net benefit which directly addresses the hypothesis posed by cost-effectiveness analysis. However, it is also argued that classical statistical inference (and its Bayesian counterpart) is arbitrary and irrelevant to clinical decision making. Accepting the null hypothesis when a new treatment has a positive but statistically insignificant mean incremental net benefit imposes unnecessary costs which can be valued in either monetary or effectiveness terms. Decisions should be based only on the posterior mean irrespective of the level of significance or whether it falls outside a Bayesian range of equivalence. Although the variance of net benefit is irrelevant to the choice of treatment alternatives, it is relevant to the question of whether more information should be acquired.

A Bayesian decision theoretic approach is presented which provides a measure of the maximum value that can be placed on additional information. If the fixed cost of research is less than this maximum, then acquiring additional information is potentially cost-effective. Based on an explicit valuation of the marginal benefit and marginal cost of sampling, the optimal allocation of trial entrants and the optimal sample size can be determined. This provides the necessary and sufficient conditions for deciding to acquire sample information and ensures technical efficiency in research design. Once additional sample information is available the treatment decision can be revised taking account of any sunk costs associated with switching between treatments. This approach has implications for the regulation of new technologies and setting priorities in research and development. However, the key conclusion is that classical statistical inference (and its Bayesian counterparts) should be abandoned in favour of an approach that can directly address the decisions that economic (and clinical) evaluation claims to inform.

Section snippets

An appropriate test statistic for CEA

The literature which considers the statistical properties of cost-effectiveness analysis has focused on estimating the sampling distribution of incremental cost-effectiveness ratios (e.g., O'Brien et al., 1994; O'Brien and Drummond, 1994; Wakker and Klaassen, 1995; Willan and O'Brien, 1996; Mullahy, 1996; Mullahy, 1997). This approach is problematic when the cost-effectiveness ratio can be infinite (if incremental benefits are zero), and when the effectiveness-cost ratio can be infinite (if

The irrelevance of inference

Whether this test statistic is at all relevant to the decision of which programme should be adopted is not clear. It has been argued for some time that the probability of a type I error in a pragmatic clinical trial (which wishes to inform clinical decision making) is entirely irrelevant (e.g., Schwartz and Lellouch, 1967). Moreover it is argued here that the rules of classical statistical inference and its Bayesian counterpart (e.g., Spiegelhalter et al., 1994) are arbitrary, are inconsistent

The decision to acquire additional information

The distribution of net benefits is entirely irrelevant to the choice between mutually exclusive alternatives but is relevant to the decision of whether to collect more information to inform this treatment choice now and in the future. A decision making approach is presented. This can distinguish the simultaneous but conceptually separate steps of deciding which alternatives should be chosen, given existing (prior) information, from the question of whether more information should be acquired.

Discussion

The approach outlined above demonstrates that a Bayesian decision theoretic approach to the stochastic evaluation of health care technologies is necessary and feasible. Decisions should be based only on the mean net benefit irrespective of whether differences are statistically significant or fall outside a Bayesian range of equivalence. The distribution of the incremental net benefits is relevant only in deciding whether it is worth collecting more information about the decision problem. This

Acknowledgements

I would like to thank two anonymous referees, Tony Culyer, Andrew Jones, Larry Lacey, Kate Mayer, Aaron Stinnett, Andrew Street and Steve Walker for comments on previous drafts. I would like to acknowledge the support provided by the Harkness Fellowships of the Commonwealth Fund of New York and the Harvard Center for Risk Analysis.

References (59)

  • J.A. Bather

    Randomised allocation of treatments in sequential experiments

    Journal of the Royal Statistical Society B

    (1981)
  • J.A. Bather

    On the allocation of treatment in sequential medical trials

    International Statistical Review

    (1985)
  • D.A. Berry

    A Bernoulli two-armed bandit

    Annals of Mathematical Statistics

    (1972)
  • D.A. Berry

    Modified two-armed bandit strategies for certain clinical trials

    Journal of the American Statistical Association

    (1978)
  • D.A. Berry

    Discussion of the paper by Spiegelhalter, Freedman and Parmar

    Journal of the Royal Statistical Society A

    (1994)
  • Claxton, K., 1997. The value of clinical information: an economic approach to research priority setting. D. Phil....
  • Claxton, K., Posnett, J., 1996a. The role of decision analysis in structuring the economic evaluation of sequential...
  • K. Claxton et al.

    An economic approach to clinical trial design and research priority setting

    Health Economics

    (1996)
  • Collins, R., Peto, R., Gray R., Parish, S., 1996. Large scale randomised evidence: trials and overviews. In:...
  • T. Colton

    A model for selecting one of two medical treatments

    Journal of the American Statistical Association

    (1963)
  • T. Colton

    A two-stage model for selecting one of two treatments

    Biometrics

    (1965)
  • Culyer, A.J., 1991. The normative economics of health care finance and provision. In: McGuire, A., Fenn, P., Mayhew, K....
  • R. Deber et al.

    Using explicit decision rules to manage issues of justice, risk and ethics in decision analysis: when is it not rational to maximize expected utility?

    Medical Decision Making

    (1990)
  • B. Freedman

    Equipoise and the ethics of clinical research

    New England Journal of Medicine

    (1987)
  • Garber, A.M., Weinstein, M.C., Torrence, G.W., Kamlet, M.S., 1996. Theoretical foundations of cost-effectiveness...
  • Hey, J.D., 1984. Decisions under uncertainty. In: van der Ploeg, F. (Ed.), Mathematical Methods in Economics. Wiley,...
  • Hoel, D.G., Sodel, M., Weiss, G.H., 1975. A survey of adaptive sampling for clinical trials. In: Elashoff, R. (Ed.),...
  • N. Johnson et al.

    At what level of collective equipoise does a clinical trial become ethical

    Journal of Medical Ethics

    (1991)
  • J.M. Lachin

    Introduction to sample size determination and power analysis for clinical trials

    Controlled Clinical Trials

    (1981)
  • Cited by (697)

    View all citing articles on Scopus
    1

    Currently visiting Harkness Fellow, Harvard Center for Risk Analysis, Harvard School of Public Health. E-mail: [email protected].

    View full text