Performance of verbal autopsy methods in estimating HIV-associated mortality among adults in South Africa

Introduction Verbal autopsy (VA) can be integrated into civil registration and vital statistics systems, but its accuracy in determining HIV-associated causes of death (CoD) is uncertain. We assessed the sensitivity and specificity of VA questions in determining HIV status and antiretroviral therapy (ART) initiation and compared HIV-associated mortality fractions assigned by different VA interpretation methods. Methods Using the WHO 2012 instrument with added ART questions, VA was conducted for deaths among adults with known HIV status (356 HIV positive and 103 HIV negative) in South Africa. CoD were assigned using physician-certified VA (PCVA) and computer-coded VA (CCVA) methods and compared with documented HIV status. Results The sensitivity of VA questions in detecting HIV status and ART initiation was 84.3% (95% CI 80 to 88) and 91.0% (95% CI 86 to 95); 283/356 (79.5%) HIV-positive individuals were assigned HIV-associated CoD by PCVA, 166 (46.6%) by InterVA-4.03, 201 (56.5%) by InterVA-5, and 80 (22.5%) and 289 (81.2%) by SmartVA-Analyze V.1.1.1 and V.1.2.1. Agreement between PCVA and older CCVA methods was poor (chance-corrected concordance [CCC] <0; cause-specific mortality fraction [CSMF] accuracy ≤56%) but better between PCVA and updated methods (CCC 0.21–0.75; CSMF accuracy 65%–98%). All methods were specific (specificity 87% to 96%) in assigning HIV-associated CoD. Conclusion All CCVA interpretation methods underestimated the HIV-associated mortality fraction compared with PCVA; InterVA-5 and SmartVA-Analyze V.1.2.1 performed better than earlier versions. Changes to VA methods and classification systems are needed to track progress towards targets for reducing HIV-associated mortality,

Introduction Verbal autopsy (VA) can be integrated into civil registration and vital statistics systems, but its accuracy in determining HIV-associated causes of death (CoD) is uncertain. We assessed the sensitivity and specificity of VA questions in determining HIV status and antiretroviral therapy (ART) initiation and compared HIVassociated mortality fractions assigned by different VA interpretation methods. Methods Using the WHO 2012 instrument with added ART questions, VA was conducted for deaths among adults with known HIV status (356 HIV positive and 103 HIV negative) in South Africa. CoD were assigned using physiciancertified VA (PCVA) and computer-coded VA (CCVA) methods and compared with documented HIV status. results The sensitivity of VA questions in detecting HIV status and ART initiation was 84.3% (95% CI 80 to 88) and 91.0% (95% CI 86 to 95); 283/356 (79.5%) HIV-positive individuals were assigned HIV-associated CoD by PCVA, 166 (46.6%) by InterVA-4.03, 201 (56.5%) by InterVA-5, and 80 (22.5%) and 289 (81.2%) by SmartVA-Analyze V.1.1.1 and V.1.2.1. Agreement between PCVA and older CCVA methods was poor (chance-corrected concordance [CCC] <0; cause-specific mortality fraction [CSMF] accuracy ≤56%) but better between PCVA and updated methods (CCC 0.21-0.75; CSMF accuracy 65%-98%). All methods were specific (specificity 87% to 96%) in assigning HIV-associated CoD. Conclusion All CCVA interpretation methods underestimated the HIV-associated mortality fraction compared with PCVA; InterVA-5 and SmartVA-Analyze V.1.2.1 performed better than earlier versions. Changes to VA methods and classification systems are needed to track progress towards targets for reducing HIV-associated mortality,

InTroduCTIon
The Joint United Nations Programme on HIV/AIDS (UNAIDS) aims to reduce the number of AIDS-related deaths from 1.2 million to under 0.5 million by 2020. 1 Most HIV-related deaths occur in low-income and middle-income countries (LMIC). 2 Mortality data form the bedrock of health research and policy decision-making, but there are few direct estimates of overall or cause-specific mortality among people living with HIV in LMIC. Global estimates are generated using complex mathematical models and data from several sources, including civil registration and vital statistics (CRVS) systems, seroprevalence surveys, and antenatal and antiretroviral therapy (ART) programmes. 3 The scarcity of reliable mortality data will make it difficult to track progress towards the ambitious UNAIDS targets.
The direct estimation of HIV-associated mortality is challenging. Death certificates

Key questions
What is already known? ► Verbal autopsy (VA) is proposed to be used more widely where civil registration and vital statistics systems are weak, but VA methods have not been validated against a quality reference standard for HIV-associated deaths.
What are the new findings?
► We used the leading VA interpretation methods to assign causes of death to adults with known HIV status in South Africa. ► Among HIV-positive adults, automated VA estimates of the HIV-associated mortality fraction were much lower than the probable true fraction, though a more recent version of SmartVA-Analyze (V.1.2.1) assigned a higher fraction. ► Automated VA interpretation is likely to result in underestimation of HIV-associated mortality.
What do the new findings imply?
► The VA instrument could be enhanced by the addition of an HIV/tuberculosis module; modifications to classifications systems are needed to capture all deaths among HIV-positive people and to differentiate deaths due to HIV-related immunosuppression from other causes.

BMJ Global Health
have poor accuracy, particularly in resource-limited settings, 4 and HIV-related causes may be omitted from death certificates because of stigma. [5][6][7] In countries without robust CRVS systems, estimates of cause-specific mortality are often derived, in part, from verbal autopsies (VAs), structured interviews with relatives or carers of deceased individuals, mostly conducted at health and demographic surveillance system (HDSS) sites. 8 9 The global roll-out of ART has already had an impact on all-cause mortality in areas of high HIV prevalence 10 and cause-specific patterns are certain to change as more people receive treatment, live longer, and die of causes other than HIV. 11 Recent guidelines advocating ART for all people living with HIV will likely accelerate this process, 12 but current systems do not allow for the enumeration of all deaths among HIV-positive individuals, which is needed to differentiate between deaths due to HIV-related immunosuppression and other causes. While the WHO standardised VA instrument does ask about HIV status, it did not, until recently, ask about ART. 13 VA has been used extensively in areas of high HIV prevalence, but has not been validated against a robust gold standard for HIV-related and tuberculosis (TB)-related deaths. 14 Using data from South Africa, we aimed to estimate the sensitivity and specificity of existing WHO VA questions in detecting HIV status and of additional questions in detecting ART initiation, compared with a reference standard of confirmed HIV status and ART initiation from clinical and research data; to estimate the specificity of VA interpretation methods in assigning HIV-associated CoD, compared with confirmed and VA-reported HIV status; and to compare the HIV-associated mortality fractions assigned to HIV-positive adults by different VA interpretation methods.

Parent studies
This study was nested within three large studies conducted in South Africa between 2012 and 2015. (1) 'TB Fast Track' was an open, cluster-randomised trial of empirical TB treatment in ambulatory adults with advanced HIV who were not on ART or TB treatment at the point of enrolment, conducted in 24 primary healthcare clinics (PHCs). 15 (2) 'XPHACTOR' was an interventional cohort study investigating the use of Xpert MTB/RIF in a systematic sample of HIV-positive adults attending outpatient clinics for HIV care. 16 (3) 'XTEND' was a cluster-randomised trial evaluating the impact on mortality of Xpert MTB/RIF roll-out; it enrolled HIV-positive and HIV-negative adults who had sputum sent for TB investigation at 40 PHCs. 17

HIV-negative individuals
To estimate the specificity of the VA question 'Was there any diagnosis of HIV/AIDS?' in detecting HIV status, VAs were conducted among confirmed HIV-negative adults who died in any of the five general hospitals to which TB Fast Track participants were referred. Based on estimated mortality in the three parent studies, 329 VAs were predicted among HIV-positive individuals; VA specificity in assigning HIV-associated deaths was predicted at 90% (based on a large study in Africa). 18 A target was set of 121 VAs in HIV-negative adults to achieve 95% confidence intervals (CI) of width ±5% around a point estimate of specificity of 90%.
Registers of inpatient deaths were reviewed at hospital mortuaries; all adults who died between June 2014 and October 2015 with a hospital-assigned CoD that was not explicitly HIV-associated, traumatic or maternal were included for further review. Inclusion was initially restricted to individuals aged 18-55 years to be similar in age to HIV-positive decedents, but, due to the small numbers of deaths among young people from causes other than those listed above, this was increased to 18-70 years. The name, sex, dates of birth and death and hospital identifiers of each included adult were recorded; hospital files and the National Health Laboratory Service (NHLS) database were searched for HIV test results. The HIV test date and contact details of next-of-kin, where available, were recorded for those with a negative HIV test (rapid test or ELISA) in the 1 year prior to death and no evidence of a subsequent positive test. Relatives were contacted and a VA conducted per standard procedures outlined below. data collection VAs were conducted by trained lay-interviewers at the families' homes or at other locations of their choosing, 1-12 months after death. The WHO 2012 VA instrument was used, 19 with questions added around ART and treatment for TB (online Supplementary table 1); these questions were asked only to respondents who answered 'Yes' to 'Was there any diagnosis of HIV/AIDS?' or 'Was there any diagnosis of tuberculosis?', respectively. VA interviewers were blinded to causes of death assigned by clinicians at health facilities (unless this was disclosed by the respondent during the VA interview).
Clinical data to confirm HIV status, ART initiation, TB diagnosis and TB treatment were obtained from clinic and hospital files, parent study databases and the NHLS online database. Data were available for all decedents regarding HIV status and for all HIV-positive decedents regarding ART initiation.

BMJ Global Health
data management and cause of death assignment Quantitative VA data were entered directly into an online database (Mobenzi Technologies, Durban, South Africa) through a cell phone interface; narrative data were captured on paper. Data collected from hospital files and the NHLS database were entered into an EpiData V.3.1 database (The EpiData Association, Odense, Denmark) and data from the parent studies into SQL databases (Bytes Technology Group, Johannesburg, South Africa). VA data were interpreted by both physician-certified VA (PCVA) and computer-coded VA (CCVA); The PCVA method was based on WHO recommendations, modified in line with changes made at the Medical Research Council/Wits University Agincourt HDSS site, South Africa (online Supplementary figure  1). Two physicians, blinded to all other clinical information, including information on ART obtained through VA, independently reviewed standard VA data and separately assigned CoD using ICD-10 codes. Assigned CoD were compared and, where there were discrepancies, the cases were discussed by the two physicians, aiming for consensus. If a consensus could not be reached, a third, independent physician reviewed the data; if the CoD assigned by physician 3 matched that assigned by physicians 1 or 2, it was considered the final CoD. If no consensus was reached after review by three physicians, the individual was assigned an 'indeterminate' CoD. Mortality Medical Data System (MMDS) 2011 software (ftp:// ftp. cdc. gov/ pub/ Health_ Statistics/ NCHS/ Software/ mmds/ 2011) was used to generate a single underlying CoD from multiple CoD assigned by PCVA.
For CCVA interpretation, VA data were mapped to InterVA input variables and fed into InterVA-4.03 and InterVA-5 (http://www. interva. net/), with the prevalence of malaria set to 'low' and HIV/AIDS to 'high'. InterVA cause-specific mortality fractions (CSMFs) were generated by dividing the sum of the likelihoods of each cause category by the sum of likelihoods for all causes after the calculation of the residual indeterminate component, per the InterVA user guide 20 ; estimations of specificity and individual agreement used the 'mostly likely' assigned cause for comparison. VA data were also mapped to the Population Health Metrics and Research Consortium (PHMRC) full instrument and fed into SmartVA-Analyze V.1.1.1 and V.1.2.1 (http://www. healthdata. org/ verbal-autopsy/ tools), with 'Malaria region' and 'Free text' deselected (all versions) and 'HIV region' selected (V.1.2.1 only). Narrative data were not provided to SmartVA-Analyze as they were captured on paper.
reference definitions An individual was considered HIV positive if a positive result had been recorded and HIV negative if a negative result had been recorded in the 1 year prior to death without a subsequent positive result. An individual was considered to have initiated ART if this had been recorded in a clinic, hospital or research file; estimation of treatment adherence was not possible based on the data available. Self-report was not considered sufficient evidence of HIV status (positive or negative) or ART initiation. A death due to ICD-10 codes B20-B24 (PCVA; underlying cause), 'HIV/AIDS' (InterVA) or 'AIDS' (SmartVA-Analyze) was considered HIV-associated; a death from any other cause was considered non-HIV-associated.
statistical analysis Individuals with unknown HIV status were excluded from all analyses (figure 1). For estimation of sensitivity and specificity, answers to VA questions of 'Do not know' were recoded as 'No', to match the methods used by InterVA and SmartVA-Analyze. 21 22 The sensitivity and specificity of VA questions were calculated with exact binomial 95% CI. The specificities of VA methods in assigning HIV-associated CoD were calculated compared with confirmed HIV status, as defined above, and VA-reported HIV status, to simulate situations in which clinical HIV data are not available.
Reference CoD were not available for all decedents and, because people with HIV may die from many causes, including those unrelated to HIV, sensitivity of VA methods in assigning CoD could not be estimated. Cohen's kappa (Κ) and chance-corrected concordance (CCC) were used to measure individual-level agreement between VA methods; Lin's concordance correlation coefficient (ρ C ), CSMF accuracy, and chance-corrected CSMF accuracy were used to measure population-level agreement based on two possible CoD: HIV-associated and non-HIV-associated, as defined above. In the absence of a clinical reference standard, CCVA CoD were compared with PCVA CoD. All analyses were conducted using Stata V.14. Figure 2 was developed using eulerAPE 23 and Inkscape (https:// inkscape. org) software.

ethical considerations
This substudy was approved by the human research ethics committees of the London School of Hygiene & Tropical Medicine and the University of the Witwatersrand. All decedents were assigned numeric identities to ensure anonymity during collection of clinical data and all VA respondents gave written informed consent for interview.
role of the funding source The funders had no role in the design of the study; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.

dIsCussIon
The VA question about HIV diagnosis showed moderate to high sensitivity and specificity in correctly identifying HIV status. Added questions around ART initiation were also sensitive and a high proportion of VA ART initiation dates were within 3 months of the confirmed date from clinical records. VA interpretation methods differed widely in their estimation of the HIV-associated mortality fraction among confirmed HIV-positive individuals; CCVA methods used until 2016 gave estimates that were likely much lower than the true fraction. All VA interpretation methods showed high specificity in assigning HIV-associated CoD.

estimating HIV prevalence and ArT initiation
The moderate-to-high sensitivity and specificity of the HIV question in detecting HIV status seen here and in a study conducted in Malawi (sensitivity 83%, specificity 98%) 24 suggests that VA may be useful in generating estimates of HIV prevalence among deceased individuals. However, sensitivity was lower when tested in a larger, more diverse population: a study conducted across five sub-Saharan African countries from 1990 to 2011 focused primarily on estimating the specificity of InterVA-4 in diagnosing HIV-associated CoD 18  (n=303). 24 The consistently high specificity suggests that VA is unlikely to overestimate HIV prevalence in high prevalence settings; further evaluations of the VA question are needed, including reanalysis of existing raw VA data from HDSS sites, to assess better its suitability for estimating HIV prevalence in different contexts. The only other study to have evaluated VA for ART use estimated sensitivity and specificity at 92% and 46%, respectively (Malawi, 2009-2014, n=154). 24 The low specificity seen also in our study suggests a need to further refine questions around ART; HIV-positive individuals are often prescribed several different drugs and confusion among VA respondents is understandable. Inclusion of variants of these questions in future VA validation studies is recommended.

estimating HIV-associated mortality
The high specificity of PCVA and InterVA in assigning HIV-associated CoD is consistent with previous evaluations in settings with high HIV prevalence. The specificity of PCVA has been between 89% and 99% in studies conducted in Uganda, 25 Tanzania, 26 and Malawi, 27 and InterVA-4 has been reported as 76%-90% specific in studies in several African countries. 18 27 28 No similar evaluations of SmartVA-Analyze were found in the literature.
The consistently high specificity of VA interpretation methods suggest that they are unlikely to overestimate HIV-associated mortality. Estimating the sensitivity of a VA method, however, is challenging even when highquality data are available, given the inherent uncertainty involved in assigning CoD. Only one study, conducted by our group in South Africa, has compared VA to CoD derived from pathological autopsy in HIV-positive individuals and found that VA methods likely underestimated HIV-associated CoD. 29 In part because of difficulties faced with ICD coding, there are few direct (autopsy or clinical) estimates of the HIV-associated mortality fraction among HIV-positive adults. A recent systematic review, which included only studies of HIV-positive individuals entirely or mostly on ART (n=19 studies), estimated that 18.5%

BMJ Global Health
(95% CI 13 to 24) of deaths in HIV-positive individuals on ART in sub-Saharan countries were due to 'non-AIDS' causes (ie, 76%-83% due to AIDS). 11 This review, however, treated deaths due to ART toxicity as 'non-AIDS', which is contrary to WHO guidance 30 and means that overall HIV-associated mortality may have been underestimated.
In populations containing fewer individuals on ART, the HIV-associated mortality fraction is likely to be even higher, as seen in several pathological autopsy studies. 31 In our study, only PCVA and SmartVA-Analyze V.1.2.1 estimated an HIV-associated mortality fraction close to the figure above (80% and 85%). InterVA-4.03, InterVA-5, and SmartVA-Analyze V.1.1.1 estimated smaller HIV-associated mortality fractions (45%, 55%, and 23%); as a high number of individuals in our study had not initiated ART, these estimates are likely lower than the true HIV-associated mortality fraction in this population.
Plans are underway to integrate VA into CRVS systems, 32 a proposal made feasible, in part, by the increased efficiency of CCVA methods. VA has been previously used mainly at HDSS sites, where, through trend analysis, systematic errors in estimates could be detected and adjusted for. If used to supplement CRVS data, however, VA findings may directly influence policy and health planning decisions and will need to be more stringently validated for key causes of mortality such as HIV.
In our study, SmartVA-Analyze V.1.2.1 assigned a much higher HIV-associated mortality fraction than V.1.1.1, though the reason for the change and the effect of this on assigning other causes is not clear, as code for the software is not available. It may be that the newer version is guided by reported HIV status: V.1.2.1 assigned an HIV-associated cause to 95% of individuals reported HIV-positive in the VA interview, compared with 23% assigned by V.1.1.1 (data not shown). None of the VA methods evaluated here was provided with data on ART use; simply assigning an HIV-associated cause to all HIV-positive individuals is unlikely to be a viable long-term strategy in areas with high ART coverage. Nevertheless, both older CCVA methods likely allocate insufficient weight to the question about HIV diagnosis, which has now been shown to be sensitive and specific in Malawian 24 and South African contexts. InterVA assigns CoD based on a probabilistic algorithm, which weights questions per the recommendations of a panel of expert physicians, convened in 2006 and 2011. 33 In our study, physicians reviewing VA data gave the HIV question high importance, assigning an HIV-associated CoD to 84% of the 306 individuals reported HIV-positive and to 52% of the 42 individuals where the respondent answered, 'Do not know' (compared with 49% and 24% by InterVA-4.03; 59% and 33% by InterVA-5; 23% and 21% by SmartVA-Analyze V.1.1.1; and 95% and 7% by SmartVA-Analyze V.1.2.1 [n=459; data not shown]). In part, this is likely because all 'Do not know' answers are converted to 'No' by these CCVA methods. Along with adjustments to weighting, an ability to differentiate between the two answers may be needed in the interpretation of key questions such as this.
limitations and strengths All individuals included in this analysis were recruited from health facilities (clinics or hospitals) and around 76% died in hospitals (data not shown). This is higher than the proportion of facility-based deaths in South Africa (43% in hospital, 23% at home, 23% in unknown locations in 2016) 34 and likely in many other countries in sub-Saharan Africa (though these data are not available due to the absence of robust CRVS systems in most of these countries). 7 This may have led to overestimation of the sensitivity and specificity of questions around HIV diagnosis and ART, as families of individuals who received medical care prior to death are more likely to be familiar with medical terminology. 35 In addition, 72% of individuals were enrolled or died in urban areas; this limits the generalisability of our findings to more rural parts of South Africa or other countries in sub-Saharan Africa, which have larger rural populations (an estimated 62% of people in the WHO AFRO region were living in rural areas in 2013). 36 However, the high sensitivity and specificity of the VA HIV question in the Malawian study cited above, which was conducted in an HDSS site, 24 and the only slight-to-moderate reductions, in our study, in the sensitivity and specificity of VA questions and VA methods (specificity only) in rural versus urban settings (online Supplementary tables 2-4), suggest that these questions and methods may perform similarly in other sub-Saharan African settings, though further evaluation is needed.
The relatively low number of HIV-negative individuals reduced the accuracy of specificity estimates and, despite best efforts, the HIV-positive and HIV-negative groups were not age matched. HIV is generally less likely to be considered as a CoD in older individuals 37 (though recent data from South Africa suggest this may no longer be appropriate) 38 ; the higher median age of HIV-negative decedents may have led to overestimation of specificities in detecting HIV status and assigning HIV-associated CoD. Estimates of sensitivity, however, should not have been affected. Physicians assigning CoD were aware that decedents were likely to have been enrolled in studies of HIV/TB and this may have led to their assigning higher numbers of HIV-associated CoD; VA interviewers were aware of a decedent's HIV status before the VA was conducted, and this may have affected the way questions around HIV and ART were asked or interpreted during the interview. The most recent WHO VA instrument 39 was not used for data collection and SmartVA-Analyze was not provided with narrative data, which may have led to suboptimal performance of the software, although the omission of these data will likely have had minimal effect, as the software does not consider the phrases 'HIV' or 'AIDS' as 'words of interest'. 22 Clinical reference CoD were not available for all decedents and the sensitivity of VA methods in assigning HIV-associated CoD could not be assessed.
This study's strengths include having confirmed HIV status for all decedents included in the analysis; the BMJ Global Health median time from HIV-negative test to death being 2 weeks, with 75% of HIV-negative individuals tested less than 2 months before death; and using the leading VA interpretation methods, including the most recent versions of SmartVA-Analyze and InterVA, to estimate the HIV-associated mortality fraction among HIV-positive individuals, something not previously done in VA studies.

next steps
The UNAIDS goal of less than 0.5 million HIV-related deaths by 2020 calls for an absolute reduction in events, 1 which will require methods to identify HIV-associated deaths more accurately. 14 40 Classification systems require alterations that will allow for the identification of all deaths in people with HIV and prediction of possible changes in mortality patterns through increased availability of ART; a flexible system, with shorter or partial revision cycles, would allow for more adaptability in the face of an evolving epidemic. In the short term, other methods may provide useful data for monitoring changes in mortality patterns: a recent study from Mozambique found good agreement between CoD assigned by minimally invasive autopsy (MIA) and complete autopsy. 41 42 These findings, along with previous evaluations of MIA, [43][44][45] suggest it may be useful in providing more accurate CoD data as an adjunct to surveillance at sentinel sites in LMIC.

ConClusIons
VA interpretation methods differed in their estimates of HIV-associated mortality and older CCVA methods underestimated the HIV-associated mortality fraction; estimates from PCVA and revised CCVA methods were closer to the probable true fraction. VA questions were sensitive and specific in detecting HIV status and sensitive in detecting ART initiation. The addition of an HIV/TB module to the VA instrument is recommended; future VA validation studies should be used to trial questions prior to inclusion in the standardised VA instrument. Modifications to classification systems are needed to capture all deaths among HIV-positive people and differentiate deaths due to HIV-related immunosuppression from other causes; closely aligned is a need for more accurate direct estimates of HIV-associated mortality to track progress towards goals set by UNAIDS.