Abstract

Background. Point-of-care (POC) diagnostics for syphilis can contribute to epidemic control by offering a timely knowledge of serostatus. Although accuracy data on POC syphilis tests have been widely published, few studies have evaluated broader outcomes beyond accuracy that impact patients and health systems. We comprehensively reviewed evidence and reporting of these implementation research outcomes (IROs), and proposed a framework to improve their quality. Methods. Three reviewers systematically searched 6 electronic databases from 1980 to 2014 for syphilis POC studies reporting IROs. Data were abstracted and findings synthesised narratively. Results. Of 71 studies identified, 38 documented IROs. IROs were subclassified into preference (7), acceptability (15), feasibility (15), barriers and challenges (15), impact (13), and prevalence (23). Using our framework and definitions, a pattern of incomplete documentation, inconsistent definitions, and lack of clarity was identified across all IROs. Conclusion. Although POC screening tests for syphilis were generally favourably evaluated across a range of outcomes, the quality of evidence was compromised by inconsistent definitions, poor methodology, and documentation of outcomes. A framework for standardized reporting of outcomes beyond accuracy was proposed and considered a necessary first step towards an effective implementation of these metrics in POC diagnostics research.

1. Introduction

Syphilis is an important public health issue; the latest available worldwide estimates conducted by the World Health Organization in 1999 suggested approximately 12 million cases of syphilis [1]. With 90% of infected individuals unaware of their serostatus, a lack of timely diagnosis is a driving force of the syphilis epidemic, particularly in resource limited settings where use of standard diagnostics presents important challenges [2]. Standard syphilis screening involves initial testing with a nontreponemic specific assay and confirmation with a treponemal specific assay [36]. These tests are usually expensive, must be conducted by skilled, laboratory-based personnel, and have a long turnaround time resulting in loss of patients to followup [7, 8]. Point-of-care (POC) tests therefore offer a novel, low cost, easy to use solution that provides results in a single visit, expediting linkages to confirmatory testing, treatment, and referral. Designed for settings with limited infrastructure, POC technologies are potentially transformative tools for global syphilis control [3].

Systematic reviews demonstrated the accuracy of syphilis POC tests [9, 10]. However, as advisory bodies on diagnostics have emphasized, accuracy is a necessary but not a sufficient condition for widespread uptake of syphilis POC tests [11]. Our objective is to facilitate decision-making related to introduction of syphilis POC tests for health planners and policymakers in diverse geographical settings, by synthesising evidence on outcomes beyond accuracy, referred to as implementation research outcomes (IRO). These IROs are relevant to patients, caregivers, and health systems. This systematic review presents the first comprehensive portrait of available evidence on IROs for syphilis POC tests.

2. Materials and Methods

2.1. Information Sources

We searched the MEDLINE, EMBASE, GLOBAL HEALTH, CINAHL, Web of Science, and SCOPUS electronic databases over the period from January 1, 1980, to January 31, 2014. All languages were considered and non-English articles were translated.

2.1.1. Search String

We used the following terms: (syphilis OR Treponema pallidum) AND (point-of-care OR rapid test OR rapid assays). Keywords rather than MeSH terms were used to capture recent papers not yet indexed and to be able to search multiple databases at once. No filters for diagnostic studies were applied as these have been shown not to capture all relevant studies [12].

2.2. Study Selection

Three reviewers (Yalda Jafari [YJ], Sushmita Shivkumar [SS], and Rohit Vijh [RV]), independently conducted database searches and reviewed the titles and abstracts of articles retrieved. Articles satisfying the eligibility criteria were retained for full-text screening. Study inclusion was determined by discussion between the initial reviewers who, in cases of discordance, appealed to a fourth reviewer (Nitika Pant Pai [NPP]).

2.3. Eligibility Criteria

We considered studies satisfying the following criteria.

Study Design. All studies conducted on humans or human samples reporting IROs.

Participants. Live participants of any age and risk group using whole blood or serum specimens, either from participants or serum panels.

Interventions. Studies documenting the use of syphilis POC tests.

Outcome Measures. To systematize presentation, documented IROs were tabulated according to the working definitions in a framework proposed below.(A)Preference: documented as a proportion (numerator/denominator) with 95% confidence intervals (CI). Numerator was defined as the number of study participants (i.e., patients, doctors, nurses, and lab technicians) in the study interviewed that preferred the POC test or testing strategy to the conventional reference testing strategy. Denominator was defined as the total number of participants (i.e., patients, doctors, nurses, and lab technicians) in whom the new POC test or strategy was evaluated.(B)Acceptability: documented as a proportion (numerator/denominator) with 95% CI. Numerator was defined as the number of study participants (i.e., patients, providers, etc.) who accepted the new POC test/strategy over the conventional reference test/strategy. Denominator was defined as the total number of participants that were offered the strategy. Participants who consented to study participation were offered the new test/strategy and acceptability was thereby documented.(C)Feasibility: feasibility of test/strategy evaluated in the study was documented. Feasibility of a test/strategy was defined with completion rate of implementation of POC strategy [13]. The completion rate (a misnomer) was documented as a proportion with 95% CI. Completion typically referred to completion of the test or screening procedure. This included screening with a POC test, typically followed by initiation of linkages to improve clinical management of patients. These linkages varied from confirmatory testing of POC test results to receipt of treatment based on POC results or to downstream treatment in referral clinics. Other associated metrics/factors relevant to feasibility of a strategy were patient experience related such as convenience, comfort, noninvasiveness, or ease of use of POC tests (reported as yes/no) and/or time metric related, that is, turnaround time to test results (TAT) with POC versus standard laboratory tests also included. Turnaround time to test results with POC was documented in minutes/hours. This TAT included waiting time to test result. Turnaround time to treatment initiation or time to receipt of confirmatory test results, if reported, was also documented. All time was reported in minutes/hours. Mean time with interquartile ranges or average (median) if available was also documented. Reported barriers and challenges to implementation of POC tests were abstracted and reported sometimes under feasibility.(D)Barriers and challenges: reported barriers and challenges to implementation of POCs are recorded.(E)Impact: impact was described by its different facets including (1) proportional increase in new cases picked up with the new POC strategy compared to the reference conventional strategy, (2) proportional increase in numbers that received treatment with the new POC strategy over reference conventional strategy, (3) a reduction in time to confirmatory testing with novel POC versus conventional strategy, (4) an increase in the number of sexual partners notified with novel POC test versus reference conventional, (5) documentation of provision of interventions to the mother and infant (as in numbers/proportions) with same day testing and treatment linked with new POC strategy over a conventional strategy, and (6) lastly, a reduction in time to referral linkages with novel POC versus conventional strategy. Data permitting, proportions were expressed with 95% CI.(F)Prevalence: study prevalence (typically, period prevalence and, sometimes, point prevalence) was defined as a proportion, where numerator consisted of individuals who tested seropositive during the study period, and denominator consisted of the total study sample [14]. If available, 95% CI was expressed. For the purposes of the review, studies that used POC tests solely to determine prevalence of syphilis in the study sample were included in this category.

2.4. Data Extraction

Reviewers extracted relevant information independently using a standardized data abstraction form pretested on a subset of the sample. One reviewer (YJ) extracted data on 100% of papers, and second reviewers (SS & RJ) abstracted data on 50% of papers.

2.5. Summary Measures

A narrative approach was taken to synthesize outcome data. This is standard practice for systematic reviews with considerable methodological heterogeneity such as IROs.

3. Results

Of 71 full-text articles, 38 (53.5%) documented IROs. Following PRISMA guidelines [15], our study selection process has been presented in Figure 1. Of 38 articles, three studies (7.9%) were in languages other than English (2 in Spanish, 1 in Portuguese) and were translated. Thirty-four (89.5%) articles referred to studies conducted in low- and middle-income country settings. Thirty-four studies employed a cross-sectional design (89.5%), one was a case-control (2.6%), and one was a clustered randomized trial (2.6%) design. Study descriptions for IROs are available in Table 1.(A)Preference: seven articles discussed preference for POC strategies as compared to conventional strategies [1622]. In a Brazilian study [16], 60% of clinicians and 52% of patients preferred conventional testing over the POC strategy. Other studies documented preference for the POC strategy amongst nurses (68% [17]) and patients (62% [19], 66.1% [23]) over the conventional approach. In a study conducted in India, 99.3% (95% CI: 98.8, 99.8) of patients preferred testing with 3 POC tests simultaneously to detect syphilis, HIV, and hepatitis B [20]. In an Australian study [18], 79% preferred self-testing with a POC test. In a multicountry study [22] most clients expressed preference for finger prick rather than venipuncture.(B)Acceptability: fifteen studies reported on patient and provider acceptability of POC tests [16, 17, 20, 2233]. The POC testing strategy was found to be highly acceptable to patients [16, 20, 23, 24, 2629, 3134], clinicians [17, 22, 26], and lab technicians [16, 26].(C)Feasibility: fifteen articles reported on the feasibility of using POC tests and test performance characteristics [4, 1620, 22, 23, 25, 32, 3438]. Completion rate of POC testing was high in two studies [20, 23], as patient and provider satisfaction were reported with POC testing in one study [38]. Clinicians and laboratory staff found use and interpretation of results easy [4, 17, 19, 22, 35, 37]. In six studies, a reduction in turnaround time to testing and obtaining results was well received [16, 18, 19, 23, 35, 38]. In Brazil, 90% of patients reported that they would wait up to 30 minutes for results [35]. Simultaneous testing with 3 singleton POC tests (i.e., syphilis, HIV, and hepatitis B) was conducted in India and a TAT of 25 minutes (range 21–27 minutes) was reported [20]. Similarly, in a study conducted in South Africa, a TAT of 15–20 minutes was reported [36]. In three other studies, patients experienced low pain and discomfort with a finger prick, in comparison to tests requiring venipuncture: 77% [38], 91.1% [18], and 68% [16]. In a multicountry study, almost all clients who received a positive POC result received treatment, ranging from 93.6% in Brazil to 103.6% in Tanzania [22]. In an article describing studies in Uganda and Zambia with a POC strategy, almost all participants (99.0% in Uganda and 95.8% in Zambia) received same day testing and treatment with the novel POC strategy [32]. Additionally, partner testing for syphilis was also performed on-site, thereby proving feasibility of the strategy [32].(D)Barriers and challenges: barriers and challenges hindered an ideal implementation of POC testing strategies and impeded feasibility. Fifteen studies provided qualitative discussions of these challenges [1620, 23, 26, 27, 29, 34, 35, 3740]. From the patient perspective, assumed pain from finger prick was a hindrance to testing [19, 23, 29], as were reports of pain or discomfort [16, 20, 40]. Patients also expressed a distrust of POC test results [19]. From the providers’ perspective, challenges included a lack of confidence in POC test results due to the inability to differentiate between past and present infections leading to overtreatment [16, 17, 19, 35]. In addition, disagreements between test readers [37], suboptimal field conditions [34], unreadable [36] or difficulty reading results [39], concerns over procurement of POC tests [38], and the time-consuming nature of offering one-stop services with POC [19, 38] posed challenges in delivery of these services. Furthermore, not all patients agreed to undergo confirmation testing [27], hampering ideal execution of testing process. In two articles evaluating POC tests in Mongolia, inadequacies of partner notification strategies for women were brought to light, raising issues such as risk of reinfection of women if partners are not treated [40] and risk of partner violence [40]. A lack of political leadership, required to ensure the success of antenatal care (ANC) programs to prevent congenital syphilis, was also highlighted as a major barrier to uptake of POC testing [26].(E)Impact: fourteen studies reported on the impact highlighting increased testing and treatment of study participants with the implementation of a novel POC screening strategy in comparison to the reference conventional screening strategy [17, 20, 22, 27, 29, 30, 32, 36, 3944]. A study in Mongolia measured impact of POC tests in a cluster randomized trial design [40]. In comparison to clinics that used conventional testing, clinics with POC testing showed a significantly increased number of women who were tested and a significantly increased number of women and their partners who were treated [40]. In a multicountry study, the increase in numbers screened compared to preimplementation of POC testing ranged from 1.0% in China to 9.2% in Zambia [22].(F)Prevalence: twenty-three articles [18, 20, 21, 2328, 3032, 39, 41, 4452] documented prevalence in various populations and estimated the seropositivity in the study sample. Prevalence estimates varied in populations and regions and varied with samples and risk profile of participants. In ANC attendants, prevalence ranged from 0% in Afghanistan [51] to 9.2% in Zambia [32]. In high-risk populations, the prevalence ranged from 0% in Guatemala [48] to 8.9% in eastern Sudan [46].

4. Discussion

Using our framework, all the studies could be evaluated for quality of their reporting. Although evidence was primarily obtained from cross-sectional studies, outcomes were incompletely defined and reported in many studies. IROs such as preference, acceptability, feasibility, barriers and challenges, and impact outcomes were not well defined. Furthermore, the quality of evidence was impaired by unclear definitions/framework for IROs for diagnostic studies. Documentation was unclear and IROs were often expressed as percentages often without confidence intervals or without a denominator. Heterogeneity of definitions, lack of reporting of study methodology, and poor quality of reporting hampered drawing of meaningful results, although the bulk of evidence was reported in favour of POC strategies.

A determined effort needs to be made to improve the quality of assessment and reporting of IROs beyond accuracy. Impact outcomes have been clearly defined by the International Initiative for Impact Evaluation [53]. Often a vast majority of studies are cross-sectional and conducted to evaluate the feasibility of a POC strategy. Patient centered outcomes of acceptability, preference, and patient experience become equally important to justify the benefits of an alternative POC strategy to stakeholders or to scale up a project once feasibility is proven. A framework for reporting outcomes that has been proposed and utilized in this review could then be used. Clear definitions and transparency of IROs will not only help improve reporting of studies and quality of research with standardized reporting, but also facilitate comparability of findings for evaluation, across all settings. Standardized reporting will allow policymakers and providers to draw meaningful conclusions for their practice and policy. Reporting results with 95% CIs should also be emphasized. These help understand and appreciate the range of variability possible for an outcome. With these improvements, research on POC diagnostics beyond accuracy will not only be more meaningful but will also positively impact public health and clinical practice [54, 55].

5. Conclusion

In view of the growing interest in the elimination of mother to child transmission of HIV and syphilis promoted by WHO and UNICEF and an ever growing need for simultaneous rapid screening tests for HIV and related coinfections for at risk populations, POC screening strategies stand to become more complex in the near future. To report outcomes and metrics, we need to standardize and emphasize precise reporting, documentation, and measurement in publications. In this paper, we have presented a framework that defines a few of the commonly reported IROs beyond accuracy in research that involves POCs in syphilis. To improve evidence on metrics, a methodological framework that standardizes measures and parameters and ensures standardized collection, reporting, and documentation of data beyond accuracy must be emphasized. This move will greatly improve the quality of outcomes reporting in POC diagnostics and allow for enhanced documentation beyond traditional “accuracy outcomes.”

6. Summary

Implementation research outcomes documented in point-of-care (POC) diagnostic evaluations for syphilis are poorly defined, evaluated, and reported across studies. We defined operational definitions and proposed a framework to help evaluate the reporting of these outcomes. This framework for syphilis POC diagnostics research will facilitate standardized collection, documentation, synthesis of outcomes, and their inclusion in policy initiatives.

Conflict of Interests

The authors have no conflict of interests to report.

Acknowledgments

This work was supported by a Knowledge Synthesis Grant from the Canadian Institutes for Health Research (CIHR, KRS 102067). Dr. Pant Pai acknowledges the current support of the CIHR New Investigator Award 2010 and CIHR Operating Grant CIHR HBF 103210. The authors also acknowledge the data abstraction and searches performed by Sushmita Shivkumar and Rohit Vijh.