The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies
Introduction
The recent discussion of the statistical properties of cost-effectiveness analysis has considered the sampling distribution of an incremental cost-effectiveness ratio for classical statistical inference. Here it is argued that the problems associated with defining a reliable test statistic for an incremental cost-effectiveness ratio can be overcome once a price per effectiveness unit has been determined. The decision rule and the test statistic can then be based on incremental net benefit which directly addresses the hypothesis posed by cost-effectiveness analysis. However, it is also argued that classical statistical inference (and its Bayesian counterpart) is arbitrary and irrelevant to clinical decision making. Accepting the null hypothesis when a new treatment has a positive but statistically insignificant mean incremental net benefit imposes unnecessary costs which can be valued in either monetary or effectiveness terms. Decisions should be based only on the posterior mean irrespective of the level of significance or whether it falls outside a Bayesian range of equivalence. Although the variance of net benefit is irrelevant to the choice of treatment alternatives, it is relevant to the question of whether more information should be acquired.
A Bayesian decision theoretic approach is presented which provides a measure of the maximum value that can be placed on additional information. If the fixed cost of research is less than this maximum, then acquiring additional information is potentially cost-effective. Based on an explicit valuation of the marginal benefit and marginal cost of sampling, the optimal allocation of trial entrants and the optimal sample size can be determined. This provides the necessary and sufficient conditions for deciding to acquire sample information and ensures technical efficiency in research design. Once additional sample information is available the treatment decision can be revised taking account of any sunk costs associated with switching between treatments. This approach has implications for the regulation of new technologies and setting priorities in research and development. However, the key conclusion is that classical statistical inference (and its Bayesian counterparts) should be abandoned in favour of an approach that can directly address the decisions that economic (and clinical) evaluation claims to inform.
Section snippets
An appropriate test statistic for CEA
The literature which considers the statistical properties of cost-effectiveness analysis has focused on estimating the sampling distribution of incremental cost-effectiveness ratios (e.g., O'Brien et al., 1994; O'Brien and Drummond, 1994; Wakker and Klaassen, 1995; Willan and O'Brien, 1996; Mullahy, 1996; Mullahy, 1997). This approach is problematic when the cost-effectiveness ratio can be infinite (if incremental benefits are zero), and when the effectiveness-cost ratio can be infinite (if
The irrelevance of inference
Whether this test statistic is at all relevant to the decision of which programme should be adopted is not clear. It has been argued for some time that the probability of a type I error in a pragmatic clinical trial (which wishes to inform clinical decision making) is entirely irrelevant (e.g., Schwartz and Lellouch, 1967). Moreover it is argued here that the rules of classical statistical inference and its Bayesian counterpart (e.g., Spiegelhalter et al., 1994) are arbitrary, are inconsistent
The decision to acquire additional information
The distribution of net benefits is entirely irrelevant to the choice between mutually exclusive alternatives but is relevant to the decision of whether to collect more information to inform this treatment choice now and in the future. A decision making approach is presented. This can distinguish the simultaneous but conceptually separate steps of deciding which alternatives should be chosen, given existing (prior) information, from the question of whether more information should be acquired.
Discussion
The approach outlined above demonstrates that a Bayesian decision theoretic approach to the stochastic evaluation of health care technologies is necessary and feasible. Decisions should be based only on the mean net benefit irrespective of whether differences are statistically significant or fall outside a Bayesian range of equivalence. The distribution of the incremental net benefits is relevant only in deciding whether it is worth collecting more information about the decision problem. This
Acknowledgements
I would like to thank two anonymous referees, Tony Culyer, Andrew Jones, Larry Lacey, Kate Mayer, Aaron Stinnett, Andrew Street and Steve Walker for comments on previous drafts. I would like to acknowledge the support provided by the Harkness Fellowships of the Commonwealth Fund of New York and the Harvard Center for Risk Analysis.
References (59)
- et al.
Mark Pauly on welfare economics: normative rabbits from positive hats
Journal of Health Economics
(1996) - et al.
Economic foundations of cost-effectiveness analysis
Journal of Health Economics
(1997) - et al.
On the decision rules of cost-effectiveness analysis
Journal of Health Economics
(1993) A re-examination of the meaning and importance of supplier-induced demand
Journal of Health Economics
(1994)- et al.
Explanatory and pragmatic attitudes in therapeutical trials
Journal of Chronic Disease
(1967) - et al.
Mathematical programming for the efficient allocation of health care resources
Journal of Health Economics
(1996) Sequential medical trials
Journal of the American Statistical Association
(1983)- Armitage, P., 1975. Sequential Medical Trials. Blackwell,...
The search for optimality in clinical trials
International Statistical Review
(1985)- et al.
Uncertainty and the evaluation of public investment decisions
American Economic Review
(1970)
Randomised allocation of treatments in sequential experiments
Journal of the Royal Statistical Society B
On the allocation of treatment in sequential medical trials
International Statistical Review
A Bernoulli two-armed bandit
Annals of Mathematical Statistics
Modified two-armed bandit strategies for certain clinical trials
Journal of the American Statistical Association
Discussion of the paper by Spiegelhalter, Freedman and Parmar
Journal of the Royal Statistical Society A
An economic approach to clinical trial design and research priority setting
Health Economics
A model for selecting one of two medical treatments
Journal of the American Statistical Association
A two-stage model for selecting one of two treatments
Biometrics
Using explicit decision rules to manage issues of justice, risk and ethics in decision analysis: when is it not rational to maximize expected utility?
Medical Decision Making
Equipoise and the ethics of clinical research
New England Journal of Medicine
At what level of collective equipoise does a clinical trial become ethical
Journal of Medical Ethics
Introduction to sample size determination and power analysis for clinical trials
Controlled Clinical Trials
Cited by (697)
Economic evaluation of flap fixation techniques after mastectomy: Results of a double-blind randomized controlled trial (SAM-trial)
2023, European Journal of Surgical OncologyHow Uncertainty Matters Under Risk Neutrality
2023, Value in HealthWhen should global health actors prioritise more uncertain interventions?
2023, The Lancet Global HealthAdding Value to CHEERS: New Reporting Standards for Value of Information Analyses
2024, Applied Health Economics and Health Policy
- 1
Currently visiting Harkness Fellow, Harvard Center for Risk Analysis, Harvard School of Public Health. E-mail: [email protected].