There has been a lot of debate on how to ‘generalise’ or ‘translate’ the findings of an EE to other countries’ contexts, given that there is usually a strong time constraint to make policy decisions.11 34 Over the years, there has been a lot of work around the development of different checklists, summarising the list of various factors that could potentially cause or lead to uncertainties while generalising, transferring, or adapting the results of an EE to other settings.34 Broadly these factors are summarised under three specific categories of methodological features (perspective, discount rate, approaches for direct and indirect cost assessment), healthcare system factors and population characteristics.21 34 Many of these factors, either alone or in interaction with other factors, affect an EE’s cost and valuation of outcomes, thereby influencing the ICER calculation.
Summary and interpretation of the findings
The present analysis is the first of its kind to use selected aHTA approaches for EE (including literature review, quality and transferability appraisal, and costs, outcome and price adjustments), and to assess it against the traditional full EE for measuring the validity and accuracy of the findings. In terms of direction and magnitude, the adapted cost estimates varied considerably, while the adapted QALYs did not differ much, when matched with the results of the Indian reference study.
The adapted ICERs from around 90% of the selected EEs on trastuzumab were higher from the threshold value of one-time GDP per-capita of India and showed trastuzumab to be cost-ineffective for India, which matches with the overall conclusion of the Indian reference study. However, these adapted ICERs were on an average about eight times higher than the threshold value for India. Whereas the ICER reported in the Indian reference study was only 23% higher than the threshold value. This huge difference between the adapted ICER and the threshold value shows a higher level of uncertainty and needs to be interpreted with caution.
In a hypothetical scenario with unlimited resources where any intervention found to be cost-effective will be funded as a part of HBP, use of estimates from aHTA may be acceptable. However, decisions in most LMICs are usually made on the margins in the context of limited resources or finite budget, which involves prioritisation. This implies that even if several interventions are found to be cost-effective, only a few of them may become a part of the existing HBP in a particular year, which will be based on the relative ranking of ICERs. So, having a dichotomous broad conclusion that whether a particular intervention is cost-effective or not, is insufficient for LMICs, where decisions are made in the context of limited healthcare resources. The adapted ICERs need to be in a close range of the original ICERs (of the full EE), and the relative rankings should also be in the same order (as from full HTA). So, from a broader point of view of resource prioritisation, aHTA findings may result in inappropriate allocation of resources.
Moreover, the absolute difference of ICERs generated from aHTA also need appropriate consideration. In situations where a full EE concludes an intervention to be cost-effective, with an ICER marginally (10%–20%) below the threshold, an aHTA for the same intervention could contradict the findings of a full EE and conclude this intervention to be cost-ineffective, given the variation in the ICERs generated from a full EE and aHTA, respectively. Therefore, in cases where the ICER is in close range of the threshold, use of aHTA increases uncertainty. Whereas, in cases where ICERs are on the extremes, that is either significantly below the cost-effectiveness threshold or more than 2–3 times of the threshold value, then the conclusion drawn using the aHTA findings are more likely to be in line with full EE.
In some situations, the findings from the aHTA may lead to inconclusive evidence. For example, in the case of IMRT, half of the adapted ICERs shows IMRT to be cost-effective and remaining half shows it to be cost-ineffective. In such situations, the only rationale is to undertake a context-specific full HTA, to assess the true cost-effectiveness of the intervention.
A direct adaptation of model by changing the parameters that potentially differ between the study country (for which EE was actually conducted) and the decision country (for which adaptation is being done) could address most of the factors causing uncertainties in transferability.17 49 However, most of the time, researchers do not have access to original model used in EE. Furthermore, even if the model is accessible, comprehending the intricate details and calculations that went through the complex model structure again becomes difficult and equally time-consuming as developing a new model and conducting a full HTA.
It was presumed that the adapted findings from EEs of countries with similar socioeconomic or population characteristics to India might show results which are closer to the Indian reference study. However, subgroup analysis showed that both the adapted ICER values (either in scenario I or II) as well as the benchmark cost-effective price from the selected EEs of trastuzumab, irrespective of the similarity in contextual factors, differed from the findings of the Indian reference study. Likewise, the adapted ICERs from studies that had used a similar source of effectiveness data, or discount rate as used in the Indian reference study, also did not show any specific similarity in absolute terms or direction with the findings of the Indian reference study. However, when the perspective of analysis was the same there was considerably lesser variation in the adapted and Indian reference study estimates.
Specifically, for scenario II (ie, using each of the three correction factors for cost adaptation), it was observed that the conclusion from around one-third of the selected EEs showed trastuzumab as cost-effective (online supplemental appendix 8a). Furthermore, the adapted ICERs, especially from the high-income countries’ studies, dropped down significantly by 75%, which does not match with or reflect the actual scenario. All these findings need to be interpreted with due caution. Over-adjustment of cost estimates with the use of both correction factors A and B, could be one of the reasons leading to a drastic decrease in the adapted values. Moreover, this effect was more pronounced in case of high-income countries, probably due to the fact that there is a huge difference in the GDP and health expenditure of these countries as compared with India, which is also reflected in the values of correction factor of A and B for these countries.
As estimated using price-benchmarking analysis, the adapted threshold cost-effective price for trastuzumab varied appreciably from 37% on lower-side to 89% on higher-side, when compared with the cost-effective price of ₹11 049 estimated in the Indian reference study. Although the average adapted price (and even the individual adapted price from most of the selected EEs) was on the lower side of the price estimated in the Indian reference study, large variation among the individual adapted estimates, creates uncertainties and ambiguity in ascertaining a price value at which a particular drug might be cost-effective locally. Nevertheless, evidence from price-benchmarking provides a strong imperative for price regulations and negotiations in situations where the current price is significantly above the prices estimated using the price-benchmarking analysis. However, price-benchmarking can only be used for adapting the price of pharmaceutical interventions and it is not possible to use this approach for device (or healthcare programme), because of the range of factors that goes in estimating the cost per patient in case of medical devices or programme.
A fundamental conclusion from the present analysis is that cost is difficult to adapt and hard to generalise, whereas the health outcomes are closer across settings. While the existing cost correction factors adjusted for the differences in the quantity and prices of the healthcare resources, however, the variation in other important factors such as skill mix of personnel delivering care, clinical practice variability, the level at which healthcare is delivered, extent of technical efficiency across countries were still unaccounted, which might have led to uncertainty in the adapted cost estimates. Moreover, elementary differences in the context-specific assumptions, while designing an EE and its model structure, are difficult to adjust and might lead to dissimilarities in the resource use and the associated costs. It is recommended that in future, researchers should focus on developing more refined and robust methods for predicting costs across settings.
In view of the findings from our analysis, which concluded that adapted QALYs showed lesser variation when compared with the Indian reference studies, we consider that the QALY adjustment factor is more accurate than cost correction factors and could potentially be used in future for adjusting population characteristics across settings. Furthermore, for adjusting QALYs a simpler and more direct approach could be to use the average ratio of tariff values of corresponding health states in the value set of the study country and the decision country. However, while we adjusted for the difference in the age of onset of disease, life expectancy and health status preferences, which are fundamental to the calculation of QALYs, we also recognise that there could be other factors related to source of treatment effectiveness, compliance, adherence, etc. that could potentially create uncertainty in the estimation of adapted QALYs across settings. Considering this, it is advised undertaking future research to further improve aHTA methods for the adjustment of health outcomes.
Nevertheless, our study has a few limitations. First, literature review was conducted in one database only which might have resulted in some studies being missed. However, the objective of the study was to assess the adaptability of existing EEs and comparing the adapted results to those of a full EE, and not to identify all existing EEs published for a given topic. Second, we undertook adaptation for one drug and one technology; therefore, our results might not be generalisable to other HTA types including programmatic evaluations. However, given the significant heterogeneity in the healthcare structures, context and programme delivery, it is likely that there will be greater uncertainty in applying aHTA to EE of healthcare programmes. Third, for adapting cost estimates we used national level GDP or health care expenditure estimates which may have over or underestimated the costs, especially in case of studies undertaken in HICs such as Canada and the USA where there are significant variations in the price of resources across regions. While the use of subnational or regional estimates for cost adaptation may have produced more accurate estimates, however, we believe this would not change the overall conclusion of our study findings, as the overall objective of our study was to identify whether or not the adapted findings are generalisable to Indian context. Fourth, disaggregated data on QALYs was not reported in two of the studies, and hence absolute QALYs were not adapted. However, it is believed this would not impact our overall conclusions because the adapted QALYs were relatively closer to the QALYs reported in the Indian reference study and had minimal impact on the adapted ICERs. Fifth, the conclusions for aHTA for medical devices were based only on two studies. It is recommended to further carry out more validation in the context of medical devices by adapting findings from a larger number of diverse EEs. Finally, we have assessed only the accuracy of the adapted findings for EE, by comparing the adapted results with those reported in the Indian reference study. However, it is recommended that in addition to accuracy, researchers should also try to evaluate the appropriateness and robustness of the aHTA methods including those beyond EE, in future evaluations.