Analyses of the return on investment of public health interventions: a scoping review and recommendations for future studies

Return on investment (ROI) analysis is increasingly being used for evaluating the value for money of public health interventions. Given its potential role for informing health policies, it is important that there is a more comprehensive understanding of ROI analysis within the global health field. To address this gap in the literature, we conducted a scoping review of recent research articles reporting an ROI metric for a health intervention within the public sector in any country setting. The database search was limited to literature published in English and studies published between 1 January 2018 and 14 June 2021. Uses and settings where the ROI metric is being applied, key methodological features of the calculations and the types of economic benefits included were extracted. 118 relevant studies were included within this scoping review. We found that ROI analyses of health interventions differed between those that only included fiscal savings (such as prevented medical expenses) and those which incorporated a wider range of benefits (such as monetised health benefits). This highlights the variation in the definition of ROI analyses and supports the finding that ROI analyses are used for a range of different research questions/purposes within the healthcare sector. We also found that the methodologies used in ROI calculations were inconsistent and often poorly reported. This review demonstrates that there is notable variation in the methodology surrounding recent ROI calculations of healthcare interventions, as well as the definition of ROI analysis. We recommend that ROI metrics should be carefully interpreted before they are used to inform policy decisions regarding the allocation of healthcare resources. To improve the consistency of future studies, we also set out recommended use cases for ROI analysis and a reporting checklist.


Pubmed search terms
("return* on investment*"[Title/Abstract] AND ("health"[Title/Abstract] OR "healthcare"[Title/Abstract])) AND (2018/1/1:2021/6/14[pdat]) An asterisk (*), used at the end of a word, was to specify any number of characters for variable endings of a root word BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Analyses of the return on investment of public health interventions: A scoping review and recommendations for future studies
Supporting Box 1: Health economics analysis: partial evaluations versus full economic evaluations The purpose of health economics is to inform optimal resource allocation to improve the population's health.Health economic analysis and evaluation can be defined as studies that fall under this remit and they are an integral part of the development of health policy.In the context of evaluating health interventions, there is a distinct type of health economic analysis known as full economic evaluations.
A full economic evaluation is a specific type of health economic analysis that explicitly compare both the costs (use of resources) and consequences (effects) of the interventions in question with an alternative course of action, known as the comparator 1 .Full economic evaluation includes cost-effectiveness analysis, cost-utility analysis, cost-benefit analysis, cost-minimization analysis, and cost-consequence analysis.
In contrast, a partial evaluation is a type of health economic analysis that only examines either the costs or consequences of an intervention independently (i.e. they do not look at both or relate the costs to the consequences), or a study that evaluates both the costs and consequences of an intervention but only evaluates a single course of action (a cost outcome study) 2,3 .
Importantly, without an appropriate comparator, an evaluation of the benefits and costs of a new intervention can be misleading.For example, the economic benefits of a new intervention may outweigh its costs.However, it is important to compare this to the current practice in analysis.It is possible for a new intervention to have a positive ROI when ignoring potential alternative options, however, it could still be less effective and more costly in comparison to the current practice.Therefore full economic evaluations are more useful for decision making surrounding resource allocation, as they consider both the costs and benefits of interventions, and account for alternative policy options 3 .
BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Supporting Box 2: Examples of ways health benefits were monetised in ROI studies
Valuing productivity gains: Productivity gains were valued in some ROI studies by placing a monetary value on the estimated productivity losses associated with a disease that are averted due to a health intervention.This often involved valuing a patient's time based on their wages.
Willingness to pay metrics: The willingness to pay technique is based on the premise that the maximum amount an individual is willing to pay (or sacrifice) for a given commodity is an indicator of its 'value' to them 4 .Using the willingness to pay technique in this context, some studies estimated what an individual is willing to pay for certain health benefits, consequently estimating the value of the health benefits of an intervention in monetary terms for that individual.An advantage of this is that it is argued that when an individual is considering their maximum willingness to pay, they will take account of all the attributes of the service of importance to them, not just the health gains 4 .An example of a willingness to pay metric is the "value per statistical life" (VSL) which captures how much individuals are willing to pay to reduce the risk of death and is used to estimate a monetary value on reductions in mortality (i.e. the number of deaths averted by a health intervention).This is often based on trade-offs individuals are willing to make between fatality risk and consumption.This can also be expressed as a "value per statistical life year" (VSLY).The full income approach taken by some studies combined the value people place on increased life expectancy (i.e.valuing additional life-years using VSLY metrics) to changes in a measure of national income growth (such as the gross domestic product (GDP)) 5 .
Converting health benefits metrics with a thresholds: Another method was to use a threshold to convert health metrics (such as DALY averted or QALYs gained) into a monetary value 6 .This approach can use willingness to pay metrics for the conversion.
BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s) The chosen categories were adapted from those used by Pit et al. 7 , with adding an additional category for smokingrelated diseases.
BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Analyses of the return on investment of public health interventions: A scoping review and recommendations for future studies
Supporting Table S6: Checklist for the reporting ROI studies in the health sector

Item Area Reported in section Introduction
Give the context for the study, the study question, and its practical relevance for decision making in policy or practice.* 2 Justify the purpose of the analysis, target audience and why ROI is an appropriate metric.Report all analytic inputs and parameters (such as values, ranges, references).Include a Table that lists which economic benefits are being included and explicitly how they are being valued monetarily.Clearly stating if the costs relate to fiscal/tangible benefits or not.

10
Provide a clear breakdown of the ROI stratified by the different types of benefits and stakeholders.

11
Report the absolute numbers regarding the cost and benefits and not just the summary ratio/percentage.*12 Report the results stratified by including only fiscal/tangible benefits and nonfiscal.

13
If including non-fiscal savingsavoid phrasing such as for every dollar invested generates the "US$X" value in returns.

14
Perform a sensitivity analysis and describe how uncertainty about analytic judgments, inputs, or projections affect the findings.Within this include any relevant proxy measures/methods to value the economic benefits.

Discussion/conclusion 15
Explicitly describe who the "savings" or economic benefits relate to.

16
Report key findings, limitations, ethical or equity considerations not captured, and how these could affect patients, policy, or practice.*17 Discuss the limitations associated with the proxy measures/methods to value the economic benefits.

18
Discuss the generalisability or transferability of results across different settings and over timeparticularly relating to the key parameters driving the ROI.
The following is a checklist for ROI studies in the health sector.We also recommend that economic evaluations should follow the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) recommendations 9 .Note that some of these items were adapted from the CHEERS recommendations (indicated with *).
BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Structured summary 2
Provide a structured summary that includes (as applicable): background, objectives, eligibility criteria, sources of evidence, charting methods, results, and conclusions that relate to the review questions and objectives.

Rationale 3
Describe the rationale for the review in the context of what is already known.Explain why the review questions/objectives lend themselves to a scoping review approach.

3-4
Objectives 4 Provide an explicit statement of the questions and objectives being addressed with reference to their key elements (e.g., population or participants, concepts, and context) or other relevant key elements used to conceptualize the review questions and/or objectives.

Protocol and registration 5
Indicate whether a review protocol exists; state if and where it can be accessed (e.g., a Web address); and if available, provide registration information, including the registration number.

Eligibility criteria 6
Specify characteristics of the sources of evidence used as eligibility criteria (e.g., years considered, language, and publication status), and provide a rationale.Describe the methods of charting data from the included sources of evidence (e.g., calibrated forms or forms that have been tested by the team before their use, and whether data charting was done independently or in duplicate) and any processes for obtaining and confirming data from investigators.

5-6
Data items 11 List and define all variables for which data were sought and any assumptions and simplifications made.

Synthesis of results 13
Describe the methods of handling and summarizing the data that were charted.5-6

Selection of sources of evidence 14
Give numbers of sources of evidence screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally using a flow diagram.* Where sources of evidence (see second footnote) are compiled from, such as bibliographic databases, social media platforms, and Web sites.† A more inclusive/heterogeneous term used to account for the different types of evidence or data sources (e.g., quantitative and/or qualitative research, expert opinion, and policy documents) that may be eligible in a scoping review as opposed to only studies.This is not to be confused with information sources (see first footnote).‡ The frameworks by Arksey and O'Malley (6) and Levac and colleagues (7) and the JBI guidance (4, 5) refer to the process of data extraction in a scoping review as data charting.§ The process of systematically examining research evidence to assess its validity, results, and relevance before using it to inform a decision.This term is used for items 12 and 19 instead of "risk of bias" (which is more applicable to systematic reviews of interventions) to include and acknowledge the various sources of evidence that may be used in a scoping review (e.g., quantitative and/or qualitative research, expert opinion, and policy document).BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Methods 3 Describe 4
the characteristics of the study population.*Describe the interventions or strategies/scenarios being compared and why they were chosen (the comparator or counterfactual).* 5 State the perspective(s) adopted by the study and outline why chosen.*6 State the time horizon for the study and outline why it is appropriate.*7 Report the discount rate(s) and outline why chosen.*8 Describe the specific ROI calculation being used (i.e., how is the ratio or percentage being calculated).

9
Analyses of the return on investment of public health interventions: A scoping review and recommendations for future studiesPreferred Reporting Items for Systematic reviews and Metaas a scoping review. 1

5 Information sources* 7 5 Search 8
Describe all information sources in the search (e.g., databases with dates of coverage and contact with authors to identify additional sources), as well as the date the most recent search was executed.Present the full electronic search strategy for at least 1 database, including any limits used, such that it could be repeated.for selecting sources of evidence (i.e., screening and eligibility) included in the scoping review.

Table S4 : Baseline time horizon used in the analyses investigated
Studies that reported multiple baseline time horizons have each time horizon reported separately.Some studies had time horizons in their sensitivity analysis not reported here.BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Table S5 : Reporting of key features stratified by the description of the analysis used in the study Reporting of a key feature The description of the analysis used in the study
Some studies had descriptions for analysis that fitted across more than one of the chosen categories (such as both a ROI and a CEA).In such cases, the study was counted in each of the relevant categories.Therefore, some studies are counted more than once.CBA; Cost-benefit analysis, CEA; Cost-effectiveness analysis, SROI; Social return on investment, ROI; Return on investment BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance