Ethics Statement

The mortality data used in this study were obtained from the official statistics agency of South Africa, StatsSA, collecting and providing data under the provisions of the Statistics Act of 1999. [13] To obtain mortality rates, we used population data from a publically-available electronic data source of the Actuarial Society of South Africa (ASSA). [14] Ethical clearance for research involving human participants was not sought as the datasets are anonymous and contain no identifiable information of any study participant. Ethical concerns regarding participant consent and possible negative consequences to study participants have been taken note of, but are not relevant to the study as the study ‘participants’ are deceased persons. In interactions with collaborators in the host country, however, relevant ethics considerations such as respect for local customs and legal requirements regarding data-use have been upheld.

Assessment Framework

In the past, completeness of death registration was commonly the only assessment criterion applied to evaluate the quality of national vital statistics. [3] However, as awareness of the usefulness of cause-of-death statistics increased, more assessment criteria were proposed. [15] These criteria have been expanded and used in a framework of which the origin [5], [15] and conceptual underpinnings [4], [6] have been described elsewhere. To evaluate South Africa’s mortality data, we built on earlier country-specific evaluations, [4]–[6] employing the general attributes and criteria as defined in the China study [4]:

1. Generalizability, informed by the criteria (1) coverage and (2) completeness of death registration;
2. Reliability: (3) epidemiological consistency and (4) temporal consistency;
3. Validity: (5) content validity, (6) use of ill-defined and non-specific codes, and (7) use of age- and sex-improbable classifications; and
4. Policy relevance: (8) timeliness, and (9) availability of sub-national data.

Each criterion is rated with three broadly-defined evaluation measures: “satisfactory”, “unsatisfactory” or, where the information is unavailable or insufficient, “unknown”. For differentiating between “satisfactory” and “unsatisfactory”, we employ the thresholds suggested in previous studies [4], [6].

Data Sources

For five criteria (coverage, completeness, timeliness, sub-national availability and content validity) information was reviewed in relevant legislation, statistical releases, web-based data repositories, research and government reports, and scholarly journals to inform about developments over time which shaped the current status of these criteria in terms of data adequacy.

For the remaining four criteria (epidemiological consistency, temporal consistency, age/sex classification, and ill-defined/non-specific codes) the evaluation draws on a dataset produced by StatsSA with 11 years’ mortality data from DNFs for 5.38 million deaths that occurred nationally from 1 January 1997 to 31 December 2007. [16] This dataset comprises of deaths certified according to the following practices. In cases of natural deaths with access to a medical practitioner, the 1992 Act requires the practitioner to complete a DNF (Form BI-1663). The DNF also makes provision for a registered professional nurse to do so. If neither is available, as may happen for example in remote rural areas, a Death Report (From BI-1680) must be completed by an authorized traditional leader (headman/chief), member of the police service, or funeral undertaker to certify the death and describe the circumstances that led to the death. [17], [18] Unnatural deaths are subject to medico-legal investigation in terms of the Inquests Act of 1959. On receipt of the DNF or Death Report by the Department of Home Affairs, the death is registered into the electronic civil registration system. Hereafter, the forms are collected by StatsSA where trained nosologists code all causes to ICD-10 3-digit codes. [19] Underlying causes are derived automatically with the Automated Classification of Medical Entities software (ACME 2000.05) [20].

Assessing the General Attributes and Criteria

Generalisability, or the extent to which mortality statistics are representative of the population under study, was assessed using the criteria coverage and completeness. Coverage refers to the extent of inclusion of different sectors of the population in the civil registration system, such as geographical sectors (e.g. urban/rural, or sample-based areas); administrative sectors (e.g. provinces, states or districts); or population groups based on country-specific categorizations. Completeness refers to the extent to which deaths within the covered population are reported into the civil registration system. For coverage, we reviewed and summarised the effect of legislation and policies that mandated and/or constrained geographic, administrative and population coverage of death registration over the past 150 years. Due to unrepresentativeness of the total population and the potential of introducing biases into the data, coverage of less than the total population is deemed ‘unsatisfactory’. For completeness, published estimates of under-registration of deaths were reviewed. Because of the need to measure the patterns and rates of mortality in a population with the minimum biases, completeness of less than 90% of the covered population is rated ‘unsatisfactory’.

Reliability relates to the consistency of mortality data with regard to established epidemiological expectations. For this general attribute, we evaluated two criteria: epidemiological consistency and temporal consistency. Epidemiological consistency of the South African data was evaluated using methods similar to those used in previous country evaluations of national vital registration systems, [4], [6] and a variation thereof. Based on the premise that the composition of mortality by cause changes systematically as all-cause mortality decline, [21]–[23] observed broad patterns of causes of death were compared with expected broad-cause values considering the relationship between the overall level of mortality and the relative contribution of causes to the overall level. The country’s gross domestic product (GDP) is used as a covariate in the model. Such evaluation is based on the theory of the epidemiological transition, according to which declines in all-cause mortality are accompanied by shifts in proportionate mortality: in high-mortality populations, communicable, reproductive and nutritional conditions predominate, whereas chronic and degenerative conditions predominate in low-mortality populations. [21] A historical dataset of international vital registration data was analysed by Salomon and Murray [23] to develop regression models that predict cause-specific compositional mortality by broad cause groups, for given inputs of all-cause mortality by age and sex. The three broad-cause groups are (1) a combined group of communicable diseases, maternal, neonatal and nutritional causes, (2) non-communicable disease, and (3) injuries, as defined in the Global Burden of Disease 1990 study [24].

To assess epidemiological consistency, the model predictions by age, sex and broad cause were compared with observed proportions for South Africa. A difference of more than two standard deviations (>2 SD) between observed and predicted proportions suggests unsatisfactory epidemiological consistency of the observed data, unless there are plausible epidemiological reasons for such departures. [4] We used national mortality data by age and sex from civil registration for 2007; population estimates for 2007 from the ASSA2008 AIDS and Demographic Model (ASSA2008) of ASSA; [14] and 2007 GDP estimates from StatsSA [25] to derive model-predicted broad-cause proportionate mortality by age and sex. At first, we compared the broad-cause proportions derived from the cause-of-death models with observed proportionate mortality for South Africa. However, as the compositional cause of death models are based on mortality schedules from countries and time periods not affected by HIV/AIDS, we also compared the model-based predictions with observed broad-cause proportions after excluding from the observed data the large numbers of death due to HIV/AIDS for 2007 as estimated in preparation for the second South African National Burden of Disease study [26].

Temporal consistency was evaluated by examining whether proportionate mortality from 10 leading causes or cause-groups changed in a predictable manner over time in the period 1997 to 2007. This criterion is informed by the proposition that proportionate mortality from different causes changes in a predictable manner over time as overall mortality changes with socio-economic development. [21], [23] In the absence of substantial natural disasters, pandemics, or revisions to the classification of diseases, a consistent trend in cause-specific mortality should be observed. Where such impacts occurred, as in the case of the substantive HIV/AIDS epidemics in sub-Saharan Africa, observed cause-specific mortality trends would be expected to reflect increased deaths resulting from the epidemic. We investigated the trajectory over time of malignant neoplasms, ill-defined natural causes, external causes, and infectious and parasitic disease which were among the most commonly-reported categories or groups of disease during the 11-year period. We also examined tuberculosis, lower respiratory infections, diarrhoeal disease, ischaemic heart disease (IHD), stroke, and diabetes, counting among the most commonly-reported communicable and non-communicable single causes for 1997–2007 and ranking among the 10 leading single causes in the South African National Burden of Disease Study, 2000 [27].

For the attribute validity, we sought to assess the extent to which mortality data show what they purport to show, and to assess the extent of insufficiently- and inappropriately-attributed causes of death. Three criteria were assessed based on information on DNFs. Content validity (criterion 5) was assessed by reviewing local studies that examined the accuracy of cause attribution. Like inaccurate cause attribution, the use of ill-defined or non-specific codes (criterion 6) is a large impediment to local usefulness and international comparison of cause data. A proportion larger than 10% of total deaths assigned to ill-defined or non-specific codes was considered unsatisfactory. Aggregated data for 1997–2007, nationally and by province of death occurrence, were analysed to identify the extent of Chapter R codes (Symptoms, signs and ill-defined conditions); three non-specific cancer codes (C76, C80, C97); two major ill-defined cardio-vascular disease (CVD) causes (heart failure (I50) and cardiac arrest (I46)); and injuries of undetermined intent (Y10–Y34). Additionally, to compare the extent of R codes by age, R codes in each age group were calculated as a percentage of total deaths in each age group. Finally, the number of deaths coded to R codes was calculated by province for each year to compare the trajectory of R codes to that of the total number of deaths over the eleven years for each province.

Criterion 7, use of age- and sex-improbable classification, is guided by the observation that certain conditions occur primarily in specific age ranges, or cause sex-specific mortality. Departures from anticipated age/sex patterns raise concern about the quality of cause data. The aggregate dataset was examined for departures from 10 sex-specific conditions comprising maternal causes of death and genital tract cancers (Text S1). Age patterns were examined for plausibility and consistency in 27 typically age-dependent causes/cause groups: maternal conditions, perinatal conditions, 16 cancers, cardiovascular disease, and suicide (Text S1). In addition, unadjusted annual age-specific death rates were calculated over the 11-year period for three leading cause groups, i.e. cerebrovascular disease, malignant neoplasms, and IHD, to assess plausibility across age from the raw data. Patterns of age- and sex-specific rates were examined from the aggregated unadjusted deaths from cerebrovascular disease by province of death occurrence, and nationally, to assess age-consistency across the provinces.

Policy relevance was evaluated by assessing timeliness of the release of mortality data (criterion 8) and availability of sub-national data (criterion 9). These criteria, respectively, are informed by the proposition that out-of-date mortality data are of little relevance for policy and intervention purposes, and that nationally-aggregated data are insufficient to identify local health differentials and needed interventions by health jurisdiction. Timeliness was assessed by examining the time gap between the end of the reference period (year of death) and the time of publication of final tabulations. A lag of two years was considered a reasonable threshold. [4] Criterion 9 was evaluated by assessing the public availability of geographically-disaggregated data in paper and electronic reports, online data repositories, and unit record data, at least at provincial level.

Criterion 1: Coverage

Death registration, enacted since 1867, was effectively a partial process for most of the 1900s as coverage was constrained by differential registration practice based on geographical segmentation and population segregation imposed by various acts under ‘homeland’ and apartheid policies. [8], [28] During the 1990s, the ‘homeland’ ideology was abolished under democratic rule, and the country was geographically unified under one government. With the Births and Deaths Registration Act of 1992, death notification became a national, inclusive legal requirement for all people in all geographic areas. Hence, coverage is rated satisfactory.

Criterion 2: Completeness

Until the 1980s, national completeness of death registration was largely unknown. [29] However, pivotal work has been done by mortality researchers during the 1980s and 1990s, using different methods and data-sources in search of plausible results. [29]–[33] After consolidating fragmented data sources, and taking into account the heterogeneity of the age structures of the four population groups, distortions introduced into the data by the rapidly- but differentially-spreading HIV/AIDS epidemic, and administrative complications of the data, Dorrington et al. [31] estimated national completeness. Applying Bennett and Horiuchi’s Synthetic Extinct Generations Method (SEG) [34] with deaths from civil registration and the Population Register, relative to population estimates from the ASSA600 AIDS model, [35] large improvements were observed in national completeness of adult deaths, increasing from 73% in 1994 to 89% in 1999/2000. [31] (Fig. 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Estimated completeness of death reporting in South Africa: 1994–2007.

Notes: *1999 and 2004 are used as midpoints between Census 1996 and Census 2001, and Census 2001 and Community Survey 2007, respectively. **Estimates apply to ages 5–85 years. 2004 was used as midpoint for the period 2001–2007. While the source [33] includes separate GGB and SEG estimates, this graph shows an average of GGB and SEG estimates combined. GGB – Generalized Growth Balance Method; SEG – Synthetic Extinct Generations Method. Source: Compiled by the authors from: Stats SA, 2006, [41] 2009, [40] 2010; [17] Dorrington et al., 2001; [31] Dorrington et al., 2004; [32] Machemedze, 2009; [43] Dorrington & Bradshaw, 2011 [33].

https://doi.org/10.1371/journal.pone.0064592.g001

A new opportunity to estimate completeness arose in 2001 with the second all-inclusive census under democracy. [36] Using population counts from the 1996 and 2001 censuses, and inter-censal deaths from civil registration, Dorrington and colleagues [32] estimated completeness with Hill’s Generalized Growth Balance (GGB) [37] method with an adaptation to account for migration. [32] National completeness levels were estimated at 83.4% for males and 84.5% for females. [32] (Fig. 1) In 2007, another nationally-representative data source, the Community Survey, [38] offered further opportunity to assess completeness. Investigating maternal mortality in South Africa, Dorrington and Bradshaw [33] used population counts form the Community Survey and the 2001 census, and deaths from vital registration, and estimated national female completeness for 2001–2007 at 91% with the GGB method, and 89%, using the SEG method (Fig. 1). Furthermore, using the Preston and Hill method [39] with inputs from StatsSA mid-year population estimates and vital registration deaths, StatsSA has estimated completeness to have improved to levels of 87% for males and 79% for females in 2007 [40], [41].

Large improvements have been estimated for registration of childhood deaths. For the period 1996–2006, Darikwa and Dorrington [42] estimated infant death registration to have improved from 43% to 89%; registration for children 1–4 years, from 43% to 57%; and for children under 5 years, from 44% to 78% (childhood completeness not presented in Fig. 1). These estimates were derived using a multi-stage method using registered death data from civil registration, data from the 2007 Community Survey (children ever born/children surviving data, data on the survival of the last child born to women aged 12–49 years; and child deaths over the past 12 months as reported by households); 2001 Census (reported household deaths); and previous research which used mortality data based on the 1998 SADHS and 1996 Census [42].

Figure 1 shows a trend of ongoing improvements over time, starting to satisfy the 90% threshold by the early 2000s, while more recent estimates by Dorrington and Bradshaw [33] and Machemedze [43] exceed 90%. For the Global Burden of Disease Study 2010, Wang and colleagues estimated completeness in South Africa at 95% since 2000, [12] and a satisfactory rating is given for completeness.

Criterion 3: Epidemiological Consistency

Large differences were found between the observed and predicted proportional broad-cause mortality patterns for 2007 (Fig. 2A) before taking HIV/AIDS deaths out of the schedule. Considerably better congruence was found between predicted and observed proportions after removing HIV/AIDS deaths (Fig. 2B). However, close to half of male and 41% of female values still deviated by more than two standard deviations from mean predicted proportions in Fig. 2B. These findings have two interpretations. Firstly, the differences in Figures 2A and 2B indicate that HIV/AIDS mortality is not adequately predicted by the cause-of-death models used in this analysis. Secondly, the persistence of deviations after exclusion of HIV/AIDS deaths, particularly for Group 1 conditions in males, could either be a real phenomenon of higher proportionate mortality from infectious diseases, or an artefact of data quality in terms of cause-of-death assignment. On the basis of differences between the South African epidemiological profile and that represented in the model predictions, as well as the observed residual differences on exclusion of HIV/AIDS deaths, this criterion could not be assessed conclusively as the model does not have the discriminatory power to enable an assessment for South Africa.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. A. Number of standard deviations by age and sex for South Africa, 2007: HIV/AIDS deaths included.

The number of standard deviation by which observed broad-cause mortality proportions differ from mean predicted proportions when the estimated number of HIV/AIDS deaths are included in the analysis. B. Number of standard deviations by age and sex for South Africa, 2007: HIV/AIDS deaths excluded. The number of standard deviation by which observed broad-cause mortality proportions differ from mean predicted proportions when the estimated number of HIV/AIDS deaths are not included in the analysis. Source: Mortality data for 2007 from StatsSA vital registration data; [16] HIV/AIDS estimates from Bradshaw et al.; [26] population data from ASSA2008; [14] GDP data from StatsSA [25].

https://doi.org/10.1371/journal.pone.0064592.g002

Criterion 4: Temporal Consistency

Temporally consistent trends are shown over the 11-year period with no large annual fluctuations in cause-specific mortality proportions for IHD, diabetes, stroke, malignant neoplasms and ill-defined natural causes (Fig. 3). Rising proportions of selected communicable diseases are consistent over time and in line with expected epidemiological patterns associated with the HIV/AIDS epidemic and the effect of misclassified HIV/AIDS deaths. [44], [45] Given trends that are consistent and concurring with local epidemiological experiences, the data is rated satisfactory for this criterion.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Proportion of total deaths due to leading categories and causes of death, 1997–2007.

Source: Vital registration data from StatsSA [16].

https://doi.org/10.1371/journal.pone.0064592.g003

Criterion 5: Content Validity

The extent to which certification and coding of causes of death are accurate is not routinely assessed, neither has it been studied in a nationally-representative sample of deaths. However, a few studies have shed light on the accuracy of cause attribution. Most of these studies have focussed on the misclassification of HIV/AIDS deaths, and most were limited in size and geographical coverage. Westwood studied death certificates and medical records at a teaching hospital in Cape Town among 500 paediatric deaths, and found that the under-certification of HIV-related deaths was over 11%, and that a further 30% of HIV-related deaths were classified using non-specific terms that could result in inaccurate classification. [46] Grandin et al. reported that, although HIV/AIDS accounted for the leading cause of death, causing 32% of all deaths in their study for 1999 to 2003 in a pediatric teaching hospital in Cape Town, the findings are likely under-estimating the role of HIV in causing death in this setting. [47] Comparing DNFs with medical records for 683 deaths in Cape Town, Yudkin et al. estimated that 36% of deaths attributable to HIV were classified to other conditions. A further 37% were indicated by euphemisms such as immune suppression or retroviral disease. [48] In a retrospective review of 242 DNFs of deaths occurring in an academic hospital in Cape Town during 2004, Nojilana et al. [49] found that the under-reporting of HIV/AIDS was 53%. Analysing vital registration data for 2000–2001, Groenewald et al. found 61% of HIV/AIDS deaths classified to other conditions. [44] Birnbaum et al. confirmed substantial misclassification of HIV/AIDS in registered deaths for 1996–2006, suggesting that over 90% of HIV/AIDS deaths were misattributed to other causes. [45] In a recent validation study of 703 deaths in Cape Town, substantial misclassification was found not only for HIV/AIDS, but also for IHD, hypertensive disease, and diabetes. [50] Content validity is rated unsatisfactory based on these findings.

Criterion 6: Use of Ill-defined and Non-specific Causes

Nationally, 12.8% of registered deaths were assigned an R code during 1997–2007, ranging from 12.2% in 1999 to 13.9% in 2007 (Table 1). Typically, about 11% of adult (aged 15–64) deaths had R codes assigned, increasing steadily over the older age groups to double this at ages 80+ (Fig. S1A). Close to two-thirds of R codes were found in persons aged 0–64 years and 36% in ages 65+. Cause-specific proportions from ill-defined and non-specific conditions show substantial differences among the provinces (Table 1). While the number of registered deaths increased dramatically in all nine provinces over time, even doubled in six, the trajectory of Chapter R codes are very different, with much less dramatic increases in absolute numbers (Figures S1B+C). However, with over 10% of causes assigned an R code, and a quarter assigned R- and non-specific codes collectively, this criterion is rated unsatisfactory.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Percentage of total deaths assigned selected ill-defined and non-specific codes by province of death occurrence, South Africa, 1997–2007.

https://doi.org/10.1371/journal.pone.0064592.t001

Criterion 7: Use of Age- and Sex-improbable Classifications

Departures from uniquely sex-specific causes were found in less than one in a million deaths, and from age-dependent causes, in less than one in two thousand deaths (Text S1). Annual age-specific death rates from the raw data for two leading single causes (stroke and IHD) and one leading cause category (malignant neoplasms) reflect the typical increase with increasing age consistently over the 11-year period. Small inconsistencies appear at ages 80 and over for 1997–1999, possibly reflecting age misreporting, problems with cause attribution in older patients with multiple cause presentation, or lower certification and coding accuracy during earlier years of the reference period (Fig. S2A). Age and sex patterns of unadjusted stroke deaths by province appear plausible over age, though starting at relatively young ages (30 years). These patterns, however, are consistent over age for both sexes and across the provinces (Fig. S2B). Criterion 7 is therefore rated satisfactory.

Criterion 8: Timeliness

Prior to 1994, the extraction and processing of mortality data from DNFs were tedious. After recording a death on the population register, images of the DNFs were put on microfilm, microfilmed images were manually verified against the original DNFs, DNFs were manually indexed and then sent to StatsSA for processing. The verification process, in particular, could take very long. [8] With limited staff training, the classification and coding of causes of death also contributed to delays, [51] and substantial hold-ups followed in the production of mortality statistics during the 1990s. For example, up to November 2001, the latest cause-of-death data available were from 1996, and in December 2001, cause data for 1997–2000 were released. Post-democracy, however, delays have been addressed by concerted multi-stakeholder efforts with a shared common vision to enhance vital statistics; a driving process encompassing motivation and learning from expert opinion; and international donor assistance aimed at staff training in several aspects of vital statistics compilation. [8], [17] Over the past decade, further improvements followed with advances in electronic data compilation, advanced training of nosologists by Australian and United States experts, and automation of cause attribution. [8] Shorter waiting times and improved regularity followed, such that, since 2007, annual reporting was done consistently on deaths that occurred in the calendar year two years prior to publication. [52] Hence, timeliness is rated satisfactory.

Criterion 9: Sub-national Availability

South Africa has nine provinces, 46 district municipalities and 231 municipalities. While statistical releases, published in 1999 [53] and 2001, [51] contained limited mortality data by province as the lowest level of disaggregation, recent reports include numerous tabulations by province and district municipality. At provincial level, web-based tables can be created with a number of DNF variables, including underlying cause. The DNF provides information on the place of death occurrence, place of registration of the death, and place of usual residence of the deceased, holding potential for a wide range of mortality and epidemiological analysis at sub-national level. Anonymous unit-record cause data by province are available electronically upon request, subject to data-use agreements. This criterion is thus rated satisfactory.

Study Limitations

The study has limitations. The intended evaluation regarding epidemiological consistency could not be assessed conclusively as the model does not have the discriminatory power to enable an assessment under this criterion for South Africa, and the rating for this criterion remains unknown. The sub-national data made available for our analysis are classified by province of death occurrence. While such data enables an understanding of data quality at the point or area of death, it also limits an understanding of mortality patterns at the place of usual residence of the deceased in those cases where the place of death occurrence differs from that of usual residence. With no national validation studies available, we based our unsatisfactory rating for content validity on results from mainly small-scale studies, and studies with a focus on HIV/AIDS. It may therefore be argued that this criterion cannot be evaluated appropriately, and should be rated ‘unknown’ rather than ‘unsatisfactory’.

Conclusion

Improvements over time, and six satisfactory ratings out nine quality assessment criteria, are a tribute to the focused efforts and investments by strategic players in the death registration system over the past two decades. Performance of criteria assessed in earlier evaluations [1], [3] have improved such that it would shift South Africa’s mortality statistics from “low” to “medium-high quality”, using the evaluation categories of Mahapatra et al. [3] However, the criteria rated ‘unsatisfactory’ have substantial potential to influence the reliability and validity of mortality data, and point to cause-of-death attribution, in particular, as a vulnerable component in producing good-quality local mortality data.

In previous studies, [1], [3], [61] the low availability of mortality data, weak quality of all-cause mortality data, and the absence of single-cause data from sub-Saharan African countries often stood out. This analysis symbolizes optimism for a region marred by data absences and deficiencies by showing considerable progress from a system paralyzed by disintegration, disparity and delay to one with potential to offer integrated, inclusive and timely sub-national all-cause mortality statistics that could be adjusted and used for demographic and health analysis. Additionally, cause-of-death data, certified and coded according to international standards in a recent ICD revision, are available by regional and socio-demographic disaggregation. However, a considerable confidence gap remains for single-cause data to be used for health planning without skilfully estimating adjustments for biases. This assessment, and other research, [26], [71]–[74] have identified priority actions that the South African civil registration, health, and education and training authorities need to take to significantly improve confidence in its mortality statistics, in particular, to improve the accuracy of national single-cause data. Improving the accuracy of single-cause data will be a critical contribution to the epidemiologic and population health evidence base in Africa.