Use of cross-reactive serological assays for detecting novel pathogens in wildlife: Assessing an appropriate cutoff for henipavirus assays in African bats

Highlights

• We assess microsphere binding assay cutoffs in the absence of gold standard assays.

• Binding assay output on a continuous scale was visualised in frequency distributions.

• Bimodal distributions with overlapping tails suggested positive and negative serogroups.

• Bayesian analyses allowed identification of an optimal cutoff from distributions.

Abstract

Reservoir hosts of novel pathogens are often identified or suspected as such on the basis of serological assay results, prior to the isolation of the pathogen itself. Serological assays might therefore be used outside of their original, validated scope in order to infer seroprevalences in reservoir host populations, until such time that specific diagnostic assays can be developed. This is particularly the case in wildlife disease research. The absence of positive and negative control samples and gold standard diagnostic assays presents challenges in determining an appropriate threshold, or ‘cutoff’, for the assay that enables differentiation between seronegative and seropositive individuals. Here, multiple methods were explored to determine an appropriate cutoff for a multiplexed microsphere assay that is used to detect henipavirus antibody binding in fruit bat plasma. These methods included calculating multiples of ‘negative’ control assay values, receiver operating characteristic curve analyses, and Bayesian mixture models to assess the distribution of assay outputs for classifying seropositive and seronegative individuals within different age classes. As for any diagnostic assay, the most appropriate cutoff determination method and value selected must be made according to the aims of the study. This study is presented as an example for others where reference samples, and assays that have been characterised previously, are absent.

Introduction

Serological assays are a valuable and widely used tool for studying infectious disease ecology in wildlife. However, inferences from assay results often are made based on a number of assumptions that may, or may not, be fully justified (for review, see Gilbert et al., 2013 Ecohealth in press). For example, it may be assumed that a diagnostic assay can “discriminate two mutually exclusive states of tested animals” (Greiner et al., 2000) (e.g. individuals are either ‘seropositive’ or ‘seronegative’). In fact, there is likely to be considerable overlap between these two states due to the dynamic nature of infections and antibody responses within individuals and across populations. An assay cutoff therefore must be selected which artificially dichotomises the antibody response observed into positive and negative results and achieves the desired sensitivity and specificity of the assay according to the needs of the study.

The complexities of interpreting serological results are compounded when the agent being studied is novel and unknown and, in the absence of specific diagnostic assays, existing assays often are used outside their original scope. This is particularly the case in wildlife disease research, where serological cross-reactivity to known pathogens may be detected within a new species or a new geographic area well in advance of detection or isolation of the actual pathogen(s). In some cases, it may be many years or decades before the causative agent is definitively isolated and characterised from the wildlife host (e.g. Hendra viruses in Australian bats (Halpin et al., 2000), Ebola virus in African fruit bats (Bossart et al., 2005, Bossart et al., 2007, Leroy et al., 2005, Li et al., 2008)). In the meantime, valuable information can be obtained using existing assays to which there is cross-reactivity and/or cross-neutralisation, providing the limitations of the assay are recognised and inferences based on results are made with caution (for example, Hayman et al., 2012).

Development and validation of diagnostic assays is recommended (Jacobson, 2009), a process which determines “the fitness of an assay, which has been properly developed, optimised and standardised, for an intended purpose”. However, full validation of an assay for use with a novel pathogen is impossible if the pathogen is yet to be definitively identified and known positive and naïve control samples are unavailable. This is also the case when an existing assay is used with samples from alternative species (Gilbert et al., in press). In the meantime, attempts should be made to determine the validity and limitations of using pre-existing assays across the species and pathogen boundaries, including comparison against alternative assays which may detect antibodies in different ways (e.g. antibody binding and neutralisation assays) and assessing assay performance across populations and laboratories.

An appropriate threshold, or cutoff, against which samples can be designated as ‘positive’ or ‘negative’, must be determined by following logical and repeatable methods. Multiple methods are available to determine an appropriate cutoff, however the majority of these assume that known positive and naive reference samples are available. Gardner et al. (2010) reviewed statistical approaches for the evaluation of diagnostic assays in the presence and absence of available gold standard assays (one that assumes near-perfect classification of infection status). In the presence of a gold standard assay, the approaches reviewed included examining diagnostic sensitivity and specificity using receiver operating characteristic (ROC) curves and likelihood ratio tests. In the absence of a gold standard assay, Bayesian or maximum likelihood latent class models were cited as powerful approaches that enable the sensitivity of two assays being compared to be estimated jointly, without the need to assume that one is ‘perfect’. However, latent-class models are not recommended for use in comparing assays for acute infections (Branscum et al., 2005) due to ambiguity in interpreting the latent class. Additionally, if the two assays are conditionally dependent (e.g. both measure similar biological processes), then accurate estimation of the sensitivities and specificities of the tests—when used in combination—require additional parameters (the covariances between the test outcomes) to be accounted for (Gardner et al., 2000). The latent-class model approach therefore still relies on one assay being sufficiently well-characterised to provide informative priors. Where these values are unknown, as is the case when utilising existing assays for novel and unknown pathogens, the relative sensitivities and specificities of the two assays are unidentifiable.

Hendra (HeV) and Nipah (NiV) viruses (genus Henipavirus, family Paramyxoviridae) are highly pathogenic, recently emerged viruses with Chiropteran host reservoirs in Australasia (Wang et al., 2000). HeV and NiV soluble G (sG) proteins have been developed and used in highly sensitive multiplexed microsphere binding and inhibition assays on the Luminex^® platform (Luminex, Austin, USA), allowing high-throughput multiplexing and, as with ELISA assays utilising the same sG proteins, allowing detection of HeV and NiV antibodies without the requirement of BSL4 laboratories for neutralisation assays (Bossart et al., 2005, Bossart et al., 2007, Li et al., 2008). While related henipa- or henipa-like viruses have been detected serologically or by PCR in mainland Africa (Hayman et al., 2008, Hayman et al., 2011, Drexler et al., 2009, Drexler et al., 2012, Baker et al., 2012, Peel et al., 2012, Weiss et al., 2012), no associated virus has been isolated to date and therefore no specific serological assays have been developed. The HeV and NiV sG proteins were found to elicit highly cross-reactive humoral immune responses to known henipaviruses, and the multiplexed assays have therefore been used to screen African bat serum and plasma samples for henipavirus antibodies (Hayman et al., 2008, Peel et al., 2012). While current assays must be used with caution, they have helped improve understanding of the distribution and dynamics of African henipaviruses (Hayman et al., 2008, Peel et al., 2012) until such time that isolates are obtained and specific diagnostic assays developed.

The output values of microsphere binding assays, median fluorescence intensity (MFI), represent intensity of antibody binding on a continuous scale. A previous study reporting henipavirus antibodies using HeV and NiV microsphere binding assays reported raw data without calculating seroprevalences (Peel et al., 2012). While this avoids the difficulties associated with defining a cutoff, presentation of data in this form can cause difficulties and the ability to simplify the data into seroprevalences has its advantages. In other African studies using these assays, in addition to reporting raw MFI values, three times the mean MFI of negative bat or pig sera was used as a threshold (i.e. cutoff) for positive reactivity for the binding assay and sera with an MFI >200 were considered positive (Hayman et al., 2008, Hayman et al., 2011). The same equipment, assay and calculation for cutoff has been used for serological studies in Australasian Pteropus spp. (Plowright et al., 2008, Breed, 2010), although the MFI values and cutoff values used were not reported. It is unclear whether this ‘three times negative’ cutoff is statistically justified, or whether it is valid to apply it across multiple species or across different cross-reactive viruses.

The choice of cutoff has obvious impacts on calculated seroprevalences and therefore interpretation of the data. Standardised approaches, justification of the cutoff chosen, and/or reporting of raw data are required to allow comparisons across studies. In this study, the ultimate objective of determining a cutoff was to enable estimation of henipavirus seroprevalence in Eidolon helvum across multiple sampling events and locations, and in some cases to determine the probability of an individual animal being seropositive. Here, the cutoffs for henipavirus microsphere binding assays for E. helvum fruit bat plasma, generated from multiple methods, were compared. For each of the different cutoffs generated, a fitted mixture model was used to assess the probability of an individual being seropositive or seronegative at that value. The results indicate that the choice of method used, and cutoff chosen, is context-dependent. This study is presented as an example for other studies where reference samples, and assays that have been characterised previously, are absent.