Empiricism in non-communicable disease mortality measurement for the Asia-Pacific: lost in translation

Introduction

Non-communicable diseases (NCDs) have emerged as a major component of disease burden globally. Cardiovascular diseases, cancers, diabetes and chronic respiratory diseases are the most prevalent NCDs, accounting for 62% of global deaths in 2016.1 The United Nations Sustainable Development Goals (SDGs) for 2030 include a target to reduce this burden by 33% before 2030.2 The NCD mortality rate, defined as the unconditional probability of dying between ages 30 and 70 years from these four disease categories, is the recommended indicator for monitoring progress towards this target.3 Measuring this indicator requires accurate population-based data on causes of death, optimally sourced from national Civil Registration and Vital Statistics (CRVS) systems.4

The United Nations Economic and Social Commission for the Asia-Pacific (UNESCAP) Region covers 49 countries representing 59% of the world’s population, and accounting for 63% of the global mortality burden from these four categories of NCDs.5 6 The UNESCSAP covers five broad subregions across the Asia-Pacific, and in terms of geographical scope, includes countries from four of the six global regions served by WHO.7 Given this large population coverage as well as the magnitude of disease burden, reducing NCD mortality is a priority for improving national population health in the Asia-Pacific, and also at the global level.

In addition to variations in national population size, there is considerable national-level diversity in environment, socioeconomic development, cultural practices, health systems design and health system access, both within and across the UNESCAP subregions. There are well-known associations between these characteristics and population level epidemiological patterns of NCD burden, mediated through a range of biological and behavioural exposures at the individual level.8–10 Reliable information on population distributions of these exposures, along with accurate measurement of NCD incidence, prevalence and mortality, are necessary to guide national health programmes to reduce the magnitude of NCD burden.11 This article examines currently available information on national NCD mortality trends and differentials for UNESCAP countries, as a basis for understanding national baseline mortality levels at the commencement of the SDG period. This article also describes the information cascade regarding the data sources, data availability and analytical methods used by different international agencies that report national NCD mortality trends for UNESCAP countries, and makes recommendations to improve the empirical evidence for national mortality measurement in the Region.

NCD mortality trends in UNESCAP countries

The UNESCAP has taken a key commitment to support the United Nations Sustainable Development Agenda for 2030, in terms of statistical capacity building, monitoring, review and tracking progress towards the SDGs.12 As part of this commitment, UNESCAP has established a database and web portal to compile and disseminate information for tracking SDG targets, including the NCD mortality indicator.13 14 Table 1 presents information from the web portal on NCD mortality trends for UNESCAP countries between 2000 and 2016. There are striking sex differentials in all countries. In addition, there is a nearly 10-fold range in NCD mortality rates in 2016, from 4.7 per 1000 females in South Korea to 38.8 per 1000 males in Mongolia. At the subregional level, countries in North and Central Asia have been estimated to experience the highest NCD mortality burden. The time trends show that in most countries, there has been a minimal NCD mortality reduction during 2000–2016, which implies that considerable action would be needed to meet the SDG target of 33% reduction between 2015 and 2030.

Although these detailed time trends of NCD mortality estimates are available from the UNESCAP portal, it has been previously established that there is a critical paucity of reliable cause-specific mortality data for most developing countries, including those in the Asia-Pacific region.15 16 The limited availability of mortality data raises questions about the source of these reported NCD mortality rates. The UNESCAP SDG report states that the NCD mortality estimates were sourced from the United Nations Statistics Division (UNSD) database for SDG indicators.5 17 Following up on this reference led to an elaborate trail of information sources from different organisations that essentially present the same estimates for the NCD mortality rates as the UNESCAP SDG database. This trail comprised investigations of the UN Statistical Division database,18 WHO World Health Indicators report,19 WHO Global Health Observatory (GHO) database,20 WHO technical report on methods for cause of death (COD) assignment21 and finally the Institute of Health Metrics and Evaluation (IHME) COD database and methodology22 23 However, each information source presented different terminology and explanations for the computational approach and/or the primary data that were used to derive the published mortality rates. Figure 1 depicts this information trail, with explanatory notes for each source as follows:

Request permission

UNESCAP SDG database

The UNESCAP database describes the availability and quality of data used for deriving these NCD mortality trends in each country according to the following three category scale:5 13

1. Sufficient: An underlying data series with at least two or more data points between 2000 and 2016.

2. Insufficient: An underlying data series with only one data point between 2000 and 2016, which is insufficient to generate a historical trend.

3. No data: Indicators with no data for the country.

The report does not include any details as to the specific years for which data points were available for each country, nor does it provide any information on the use of specific data quality assessment criteria as a basis to assign countries to these categories. The UNESCAP report refers to the UN Statistics Division SDG database as its source of information for NCD mortality trends.

UNSD SDG database

The UNSD SDG database provides the same time series of NCD mortality rates for all UNESCAP countries, and all other countries in the world.17 The UNSD database provides a metadata sheet that cites a technical report from WHO as it is source for data estimation methods.21 24 The database internet website includes a link to a set of frequently asked questions, one of which provides information on the descriptors used in the database for the nature of the data and the methodology used to compute the NCD mortality rates for each country; as follows25:

1. Country Adjusted data: This description is assigned to countries for which primary data on NCD mortality is produced and provided by the country, which is then adjusted by WHO to comply with international standards and/or definitions, hence enabling international comparability of the indicators.

2. Estimated data: This description is assigned to countries for which the mortality rates are estimated using national primary data on NCD mortality derived from surveys or administrative records or other sources. WHO produces these estimates when country data for some year(s) is not available, or when multiple data sources exist for the country, or when there are issues with the quality of national CRVS data provided to WHO.

3. Modelled data: For these countries, the NCD mortality rate is modelled by WHO on the basis of other covariates, when there is a complete lack of national data on NCD mortality.

Although not mentioned, it is presumed that the primary data for ‘country adjusted’ NCD mortality rates are derived from population based CRVS systems. Similarly, it is also presumed that for some countries, the ‘estimated’ NCD mortality rates would include some input data from national CRVS systems. The UNSD database metadata sheet for the NCD mortality indicator mentions WHO GHO database as its source of information.20 24

WHO global GHO/COD report

The GHO presents the same time trend of NCD mortality rates for UNESCAP countries, among other health indicators.20 The GHO NCD mortality rates are derived from the country-level causes of death estimates developed by WHO for 2000−2016.6 The data sources and methods for these estimates are documented in a related WHO COD technical report.21 In addition to a generic description of the estimation process, the report includes an annex table that summarises the data source and method used for each country, using one of the following terms26:

1. ‘Useable VR’: This term is used for countries for which the NCD mortality rates are calculated by WHO, using the time series of useable vital registration data from their national CRVS system.21 Although not specified in this term, the actual methodology for estimating cause-specific mortality includes WHO derived specific adjustments to the country CRVS data, based on international disease-specific epidemiological models developed by WHO.27

2. ‘Global Burden of Disease (GBD) 2016+ WHO’: This term is used for countries for which WHO derived NCD mortality rates are based on the national time series of cause-specific mortality estimates that were modelled as part of the Global Burden of Disease (GBD) Study 2016, conducted by the IHME, USA. The GBD 2016 Study produced a modelled time series of mortality estimates by age, sex and cause from 1980 to 2016 for all countries.28 At WHO, these IHME GBD modelled estimates are further adjusted with findings based on disease-specific epidemiological models as mentioned earlier, and the results were used to derive NCD mortality rates.

3. ‘GBD2016adj+WHO’: This term is used to describe the methodology for China, for which the IHME GBD modelled time series of estimates for China were first adjusted using Chinese death registration data for 2013, then subsequently adjusted using WHO disease-specific models.29 These estimated deaths were then used to compute NCD mortality rates for China.

4. ‘Million Death Study (MDS)+WHO’: This term is used for India, for which annual WHO COD estimates are based on verbal autopsy data from the national Sample Registration System, which is analysed and reported under the ‘MDS’.30 The MDS data are further subjected to WHO model adjustments as above, to estimate national deaths by age, sex and cause.29 These estimates are then used to compute the time series of NCD mortality rates for India.

WHO assigns an average Vital Registration (VR) useability score (WHO VR score) to the CRVS data reported by countries over the period 2007–016, as presented in table 2. The score is based on a combination of death registration completeness and proportion of deaths with ill-defined causes in the CRVS data for each year. For China, the score applies to data from the national Disease Surveillance Point System, which covered approximately 6% of the national population during this period.21 WHO uses national data as a starting point for estimating numbers of deaths by age, sex and cause only for 70 countries which have a threshold average useability score of 60%, and which also fulfil other inclusion criteria regarding time series of data availability, and use of detailed COD lists. These include nine UNESCAP countries, as shown in table 2. For these 70 countries, the COD estimates are used to calculate the NCD mortality indicator for each reference year.

WHO World Health Statistics Report 2019

The NCD mortality indicator estimate for 2016 (from the GHO time series) is also presented for all countries in WHO World Health Statistics (WHS) Report for 2019.19 However, these estimates are described using terms that are different from those used in WHO COD Technical Report . First, all national NCD mortality rates are labelled as ‘comparable estimates’, which are defined as ‘country data adjusted or modelled to allow comparisons between countries or over time’.19 Hence, this term includes the estimates derived from all four methodological processes described in WHO COD technical report. Also, these country ‘comparable estimates’ are categorised according to the availability of primary data for computing the NCD indicator for 2016, without any indication of the estimation method used. The following categories are used:

1. Countries with primary data <4 years old.

2. Countries with primary data >4 years old.

3. Countries with no primary data.

It should be noted that as per WHO methodology, the reported NCD mortality rates in 2016 for UNESCAP countries without useable VR would have been based on the IHME GBD COD estimates.

IHME GBD COD estimates for 1980–2016

The IHME GBD national COD estimates are derived through a statistical modelling approach.28 All globally available time series of country-specific mortality data (mostly from CRVS systems but also from other sources) between 1980 and 2016 are used as model inputs. These data are incorporated into the IHME global COD database, which serves as the source for model inputs. For Bangladesh, China, India and Indonesia, data from their national sample mortality statistics programmes are included in the IHME database, since national CRVS data is not available.31–35 For each country, data availability in the database is summarised as a national ‘percent well-certified’ score for specific time intervals.28 Table 2 shows this score for UNESCAP countries during 2010–2016. This score is based on population level completeness of death recording and proportion of deaths with specific cause attribution, during the time interval. These scores are similar in concept to WHO VR ‘useability’ score reported in table 2, but cover a different time period. However, there are also differences in the use of national data for mortality estimation, according to either the IHME or WHO methods.

The IHME GBD modelling approach varies for countries according to the availability and quality of their national COD data. IHME models for countries with ‘percent well-certified’ scores less than 85% use COD data from all countries of the world (including their own national data) as inputs in the modelling process. However, for countries with scores >85%, the models exclude data from countries with lower scores from the model inputs, and only use high-quality cause-specific mortality data as inputs, in order to reduce uncertainty in the predicted death rates.28 The IHME modelling process also includes specific covariates based on national data for each country (where available), with separate sets of covariates for each COD.22 For each country, separate models are developed that predict mortality rates by sex for each COD. There are several additional estimation and adjustment models to predict mortality rates for several special causes, and these models vary by cause, location and time period.

All the predicted national cause-specific mortality rates from IHME models are first applied to national population estimates to derive an intermediate estimate of cause-specific numbers of deaths, using a procedure termed as indirect standardisation.36 These intermediate cause-specific deaths are then proportionally scaled to the demographic estimate of total deaths in each sex-age group, to derive the final IHME cause-specific mortality estimates by age and sex for each country.28 For some countries (as noted in table 2), these IHME estimates are subsequently used by WHO as a starting point in their process of adjustment and estimation as described earlier. For these countries, the derived cause-specific numbers of deaths are then used to compute the international standard NCD mortality indicator.

Comparison of national NCD mortality estimation descriptors

Table 2 presents a comparison of the descriptors used for individual countries by each data source, for NCD mortality rates that are derived from the same set of national mortality registration data, or national COD models, or both. The use of varying terminology to describe primary data availability, data useability, and/or statistical methods for each country is inconsistent. In the case of the ‘Sufficient’ rating provided in the UNESCAP Report, this is also misleading, since this is clearly not the case for most countries in the Region. Also, there is little to be gained from the descriptors used in UNSD database, since it is not possible to understand the degree to which the terms ‘adjusted’ and ‘estimated’ actually reflect the changes to the primary data, as a result of these statistical processes. For the same reason, the availability (or lack thereof) of primary data during the 4 years preceding 2016 as mentioned in WHO Statistical Report for 2019 has very little bearing on the estimated 2016 NCD rate itself, given the extensive modelling and adjustment procedures used by WHO and IHME, on any available primary data. This is further compounded by the statistical uncertainty that arises from the various steps of these modelling and adjustment procedures.

Country-level comparisons of descriptors from the different data sources show varying degrees of inconsistency. For instance, China’s NCD mortality rate from WHO WHS 2019 Report is described as based on recent primary data, which gives an impression of sufficient national data inputs from the CRVS system. However, in reality, the rate was actually modelled by the IHME, using national mortality data from a time series based on a 6% population sample, for which the data quality score was only 69%.28 29 The IHME estimates were further adjusted by WHO using Chinese national death registration data from 2013, and WHO disease-specific epidemiological models. These additional details suggest that there is very little anchorage of the published rates in the Chinese primary data, and WHS 2019 descriptor does not reflect this reality.

In the instance of India, WHO Technical Report describes the estimates to be derived from the national ‘MDS’ with WHO adjustments, conveying an impression that the published rates are derived from national data, and without using IHME models. However, the UNSD database qualifies the estimates as ‘Modelled’; which means that there was no national data available and the estimated rates were therefore derived from model covariates, as in the IHME process. At another level, although Armenia, Brunei, Iran, Kazakhstan, Russian Federation, Tajikistan, Turkey and Turkmenistan have national data that meets WHO average useability threshold of 60%, the national data are not used by WHO as they do not meet inclusion criteria related to either availability of requisite time series of data, or use of detailed COD lists for data tabulation. Therefore, WHO uses the IHME modelled estimates to derive NCD mortality rates for these countries. Finally, for large UNESCAP countries such as Afghanistan, Myanmar, Nepal, Pakistan and Viet Nam, and several other countries with smaller populations, there is no national data on causes of death, hence their rates are estimated from IHME models, but the UNESCAP report declares that the rates are based on ‘sufficient’ national data.

Discussion

Addressing the SDG target for reductions in NCD mortality requires timely and reliable data at global, regional and national levels. At the first instance, the time series of mortality rates in table 1, along with the UNESCAP rating of ‘sufficient’ primary data availability suggests that the reported data provide a reliable baseline of NCD mortality in all UNESCAP countries, for monitoring the SDG target reduction of 33% by 2030. However, the information provided in figure 1 and table 2 clearly indicate that there are critical gaps in availability and quality of cause-specific mortality data from CRVS systems in most UNESCAP countries. Most importantly though, these deficiencies are to a large extent being masked by international exercises in mortality estimation, using epidemiological models. As illustrated in this article, for many countries these models are constructed without any primary national data inputs, and for others, through complex manipulations of available data. But, the details of the methods and processes used in these estimation exercises are only available from a trail of internet sites, technical reports, scientific manuscripts, and their footnotes and appendices. Further, these details are substituted by varying cryptic descriptors across different information sources, which obfuscate the actual availability and/or quality of primary data. In all likelihood, readers of the UNESCAP SDG Progress Report 2020 would take the data availability rating of ‘Sufficient’ at face value, with only a few going on to review the rating definition, and fewer still following the complete information cascade as shown in Figure 1.

The implications of this phenomenon are manifold, and pose several challenges to the public health community. First, WHO and IHME produce separate estimates of mortality by age, sex and cause for all countries for the same time periods, which creates confusion among data users. While WHO also uses the estimates to compute the NCD mortality indicator for each country as disseminated by the GHO, the IHME does not compute or publish the NCD mortality indicator. Hence, there is no publicly available direct comparison of the measures from the two sources. We used the IHME mortality estimates to separately calculate the NCD mortality indicator for 2016 for countries with ‘Useable VR’ as noted in WHO technical report. We observed that for Fiji and Uzbekistan, the IHME estimate was higher than the WHO estimate by about 6%, while for Kyrgyzstan, it was lower by 6%, although both estimates were essentially derived from the same primary data (data not shown). These differences could be due to the IHME modelling process, or WHO adjustments from disease-specific epidemiological adjustments, or both. However, these absolute differences are not trivial for these countries, considering the implications of these baseline measurements for the degree of reductions required to meet the SDG target. Likewise, there are differences of varying extent between IHME and WHO estimates of NCD mortality rates for all countries, whether or not they are based on national primary data. These varying estimates of NCD mortality rates from different information sources create confusion at the national level as to which would be more appropriate for public health policy, and for monitoring progress towards the SDG target.37 38

Second, there are also differences between mortality indicators from primary national data and IHME estimates for individual components of the NCD mortality rate (ie, cardiovascular diseases, diabetes, cancers and chronic respiratory disease) which get masked by the comparison at the aggregate level of the composite indicator. Such differences also occur for all other causes of death, and these variations also challenge direct epidemiological inferences. For example, a recent analysis for Malaysia showed statistically significant differences between age-standardised cause-specific mortality rates for 7 out of 10 leading causes in males and females, when comparing estimates from a nationally representative epidemiological study with IHME estimates for Malaysia.39 Similar differences have also been observed for European countries with high quality mortality data, and have been the subject of debate in the international public health community.40–42 At another level too, since the IHME COD estimates are derived through indirect standardisation, the results are not directly comparable across different countries, particularly when there are differences in population age structure.36 However, the IHME GBD mortality estimates are extensively used for comparative analysis at sub national, national, regional and global levels, disregarding this limitation.43–46

Third, although this article has focused on presenting the inconsistent portrayal of data availability and estimation methodology for NCD mortality, the patterns represented in figure 1 and table 2 are applicable for all other causes of disease burden as well. The observed variations between national estimates from WHO, IHME and locally derived indicators for countries with high-quality data are a matter of concern even for countries without primary data. However, the prominent reporting of these estimates by international agencies and academic institutions in reputed scientific platforms lend an aura of credibility to these estimates for both international and national stakeholders. This aura of credibility is accentuated when the facts about data sources and methods are obscured, as demonstrated by the extensive list of references needed to establish these facts, in this article. Further, enhanced credibility also occurs from secondary analysis of these estimates at global, regional and national levels, presented in repackaged forms by various international groups with their own agenda.45 47 48 Such reports, along with detailed inferences as to their epidemiological implications, often result in their direct uptake for health policy at all levels, without any consideration as to their reliability or validity.

Finally, there is a continuous cycle of updates and release of these international estimates, often accompanied by changes to the estimated indicators for some countries for the same time periods, which are justified as resulting from additional data availability, or revised estimation methods, or both.42 In summary, for many countries, this ‘flood’ of estimates which do not have any anchorage in reliable primary data tends to divert the focus from strengthening primary data for mortality measurement.49 These estimates could be verified only through the actual collection and processing of local mortality data, aided by strengthened local capacity for data analysis and interpretation. The continued publication of these estimates suggests the urgent need for such national mortality statistics programmes.

Over the past four decades, the international community has paid increasing attention to health development. This started with WHO’s call in 1978 for ‘Health for all’ by 2000, followed by the United Nations Millennium Development Goals (UNMDGs) during 1990–2015, and the current UN SDGs agenda for 2015–2030. Concomitantly, there has been an increased requirement of information to monitor progress towards these targets, starting with life expectancy at birth and infant mortality rate to measure health status for the Health for All programme, and child, maternal and infectious disease mortality rates for the UNMDGs.50 51 These data requirements were limited, and were met to some extent through population censuses, demography and health surveys, and infectious disease surveillance programmes.52 However, the UNSDGs required detailed information to monitor a range of infectious and NCDs, as well as mortality from injuries and various social and environmental exposures.15 Well-functioning CRVS systems are a natural and optimal source of primary data for these requirements, and need to be strengthened in order to resolve the challenges in mortality measurement and interpretation as reported in this article.

The need for reliable primary data on mortality is particularly important for the UNESCAP region, given its extensive population coverage and the magnitude of potentially avertable disease burden across countries. UNESCAP has launched its regional ‘Get Everyone in the Picture’ Initiative for the CRVS decade 2015–2024.53 There is an urgent need to conduct CRVS system strengthening programmes in many UNESCAP countries with zero or low data quality scores, as shown in table 2. On reviewing the findings presented here, country officials might reflect on the status of their national vital statistics programmes, and plan the way forward to improve mortality data availability.15 At another level, the four WHO Regional Offices associated with UNESCAP, in tandem with WHO Country Offices could provide stewardship and technical guidance in strengthening the reporting of causes of death, and statistical analysis.

We propose that the global health community should facilitate countries in establishing their national mortality statistics programmes, while avoiding the distraction arising from frequent release of, and debates over, modelled mortality estimates. Individual UNESCAP countries should now develop a strategic CRVS strengthening approach customised to national requirements.15 These strategies would need to be based on CRVS functional status, availability of infrastructure for health information, and prevailing levels of data quality, and would also require adequate attention being given to building local human capacity.15 54 For countries without data, the strategy would involve a thorough CRVS situational assessment, followed by practical system design and an appropriately resourced implementation plan.15 For large countries, activities could start with a nationally representative population sample, with incremental scale up of population coverage over time.55 For countries with functional systems but problems with data quality, programmes to validate available data, to re-engineer business processes for death registration and COD ascertainment and to improve data management and analysis would be required.15

Conclusions

Reliable measurement of NCD mortality is essential for many UNESCAP countries to understand their current levels of disease burden, and plan health programmes to address the same. However, the lack of reliable national primary data limits such an evidence-based approach. The use of epidemiological models and statistical techniques to estimate NCD mortality for data-deficient countries is potentially prone to error. A clear understanding of these data limitations and potential sources of error from estimation methodology, as described in this article, should serves as a stimulus to commence programmes to improve primary data availability. Sustained activities of this nature will be required over the next decade, to translate the data descriptor for all UNESCAP countries into being truly ‘sufficient’, for measuring NCD mortality rates and other health-related SDG targets for 2030.