Background: Many decisions in health care aim to maximise health, requiring judgements about interventions that may have higher health effects but potentially incur additional costs (cost-effectiveness framework). The evidence used to establish cost-effectiveness is typically uncertain and it is important that this uncertainty is characterised. In situations in which evidence is uncertain, the experience of experts is essential. The process by which the beliefs of experts can be formally collected in a quantitative manner is structured expert elicitation. There is heterogeneity in the existing methodology used in health-care decision-making. A number of guidelines are available for structured expert elicitation; however, it is not clear if any of these are appropriate for health-care decision-making.
Objectives: The overall aim was to establish a protocol for structured expert elicitation to inform health-care decision-making. The objectives are to (1) provide clarity on methods for collecting and using experts' judgements, (2) consider when alternative methodology may be required in particular contexts, (3) establish preferred approaches for elicitation on a range of parameters, (4) determine which elicitation methods allow experts to express uncertainty and (5) determine the usefulness of the reference protocol developed.
Methods: A mixed-methods approach was used: systemic review, targeted searches, experimental work and narrative synthesis. A review of the existing guidelines for structured expert elicitation was conducted. This identified the approaches used in existing guidelines (the 'choices') and determined if dominant approaches exist. Targeted review searches were conducted for selection of experts, level of elicitation, fitting and aggregation, assessing accuracy of judgements and heuristics and biases. To sift through the available choices, a set of principles that underpin the use of structured expert elicitation in health-care decision-making was defined using evidence generated from the targeted searches, quantities to elicit experimental evidence and consideration of constraints in health-care decision-making. These principles, including fitness for purpose and reflecting individual expert uncertainty, were applied to the set of choices to establish a reference protocol. An applied evaluation of the developed reference protocol was also undertaken.
Results: For many elements of structured expert elicitation, there was a lack of consistency across the existing guidelines. In almost all choices, there was a lack of empirical evidence supporting recommendations, and in some circumstances the principles are unable to provide sufficient justification for discounting particular choices. It is possible to define reference methods for health technology assessment. These include a focus on gathering experts with substantive skills, eliciting observable quantities and individual elicitation of beliefs. Additional considerations are required for decision-makers outside health technology assessment, for example at a local level, or for early technologies. Access to experts may be limited and in some circumstances group discussion may be needed to generate a distribution.
Limitations: The major limitation of the work conducted here lies not in the methods employed in the current work but in the evidence available from the wider literature relating to how appropriate particular methodological choices are.
Conclusions: The reference protocol is flexible in many choices. This may be a useful characteristic, as it is possible to apply this reference protocol across different settings. Further applied studies, which use the choices specified in this reference protocol, are required.
Funding: This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 37. See the NIHR Journals Library website for further project information. This work was also funded by the Medical Research Council (reference MR/N028511/1).
Keywords: DECISION-MAKING; EXPERT ELICITATION; HEALTH; TRANSPARENCY; UNCERTAINTY.
Background: Decisions in health care aim to maximise health, requiring judgements about treatments. The evidence used to make these judgements is typically uncertain. In these situations, the experience of experts is essential. Structured expert elicitation collects beliefs from experts. There are different guidelines available for structured expert elicitation; however, it is not clear if any of these be can be used in health-care decision-making, for example in considering if a treatment should be made available in the NHS. This project aimed to develop a guidance for structured expert elicitation to inform health-care decision-making.
Methods: Reviews and experimental techniques were used to gather a list of methods to conduct structured expert elicitation. The suitability of these choices in health-care decision-making was then determined by comparing these with a set of standards that support the use of structured expert elicitation in health-care decision-making.
Results: Different guidelines prefer different approaches to conduct structured expert elicitation. There is a lack of evidence available to determine which of these methods is most appropriate across the whole of health-care decision-making. It is possible to define reference protocol methods that could be used in a particular type of health-care decision-making, health technology assessment. This includes gathering experts with knowledge of the clinical area, asking experts about things that they observe in clinical practice and asking experts individually for their beliefs. For decision-makers working outside health technology assessment, for example at a local level, or for treatments that are not yet available to patients, these choices may not be appropriate.
Conclusions: This flexibility of this guidance is a useful feature. It is possible for different decision-makers in health care to interpret the reference protocol for their own circumstances.