Introduction Ninety-nine per cent of all maternal and neonatal deaths occur in low-income and middle-income countries (LMIC). Prognostic models can provide standardised risk assessment to guide clinical management and can be vital to reduce and prevent maternal and perinatal mortality and morbidity. This review provides a comprehensive summary of prognostic models for adverse maternal and perinatal outcomes developed and/or validated in LMIC.
Methods A systematic search in four databases (PubMed/Medline, EMBASE, Global Health Library and The Cochrane Library) was conducted from inception (1970) up to 2 May 2018. Risk of bias was assessed with the PROBAST tool and narratively summarised.
Results 1741 articles were screened and 21 prognostic models identified. Seventeen models focused on maternal outcomes and four on perinatal outcomes, of which hypertensive disorders of pregnancy (n=9) and perinatal death including stillbirth (n=4) was most reported. Only one model was externally validated. Thirty different predictors were used to develop the models. Risk of bias varied across studies, with the item ‘quality of analysis’ performing the least.
Conclusion Prognostic models can be easy to use, informative and low cost with great potential to improve maternal and neonatal health in LMIC settings. However, the number of prognostic models developed or validated in LMIC settings is low and mirrors the 10/90 gap in which only 10% of resources are dedicated to 90% of the global disease burden. External validation of existing models developed in both LMIC and high-income countries instead of developing new models should be encouraged.
PROSPERO registration number CRD42017058044.
- systematic review
- maternal health
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
What is already known?
The development of prognostic models is a fast-growing health research field.
Prognostic models may help caregivers to guide the best treatment choices per individual patient and be more cost-effective by identifying high-risk patients who benefit most from certain interventions.
In 2016, a systematic review was published that presented all the available prognostic models in obstetrics globally (263 models based on 177 papers for 40 different outcomes; however, no differentiation was made by development or validation based on income level of country of origin.
Populations at risk and healthcare systems differ drastically between high-income and low-income countries, with the largest burden (>95%) of maternal and perinatal mortality and morbidity in low-income and middle-income countries (LMICs).
What are the new findings?
To our knowledge, this is the first systematic review that provides an overview of prognostic models for maternal and neonatal outcomes developed and/or validated in LMICs.
This review adds to and updates the previously conducted review and creates an overview of available and implementable models for healthcare providers in LMICs.
Twenty-one prognostic models were identified, most models focused on hypertensive disorders in pregnancy and only one study performed external validation of an existing prognostic model.What do the new findings imply?
What do the new findings imply?
This study provided an overview of all prognostic models developed or validated in LMICs.
Given the global distribution in burden of maternal and perinatal morbidity and mortality, this review identified a substantial research gap, with relatively few models developed and/or validated in LMIC settings.
This review can contribute to shifting the focus of current research from the development of new prognostic models towards external validation in both high-income and low-income settings and, ultimately, implementation to investigate the impact of these models in real life. This study provided an overview of all prognostic models developed or validated in LMICs.
Prognostic models can be a vital tool for the global reduction of maternal and perinatal mortality and morbidity as they facilitate timely identification of pregnant women and infants at risk of adverse outcomes, and allow for initiation of preventative or therapeutic strategies.1 2 One of the oldest and most famous prognostic models in healthcare is the Apgar score. Since its introduction in 1953, the Apgar score has been globally used by obstetricians and paediatricians for the rapid and systematic assessment of a newborns’ condition.2 3
Currently, 99% of all maternal and neonatal deaths occur in low-income and middle-income countries (LMICs).4 In order to decrease this mortality in LMICs, bottlenecks such as limited (primary) prevention of adverse outcomes, shortages of qualified staff and skilled birth attendants and unavailability of appropriate management of complications and disabilities need to be overcome.5–8 The potential impact of simple risk identification strategies, such as prognostic models and score charts like the Apgar score, to support the provision of high-quality care in LMIC settings may be substantial given these challenges.
Recently, a systematic review was published to identify prognostic models in obstetrics and their applicability.1 This review did not specifically distinguish whether the models were developed or validated in LMIC or high-income countries. It is unlikely that prognostic models developed in high-income countries are generalisable to LMIC settings where adverse outcomes occur more frequently, healthcare providers are fewer and there is less access to diagnostic or prognostic tests and treatment regimens.9 In addition, a number of new models developed or validated in LMIC have been published since. Therefore, this systematic review aims to provide a comprehensive summary of prognostic models for adverse maternal or perinatal outcomes developed and/or validated in LMICs.
Protocol and registration
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and Cochrane Handbook for Systematic reviews guided this systematic review.10 11Online supplementary data 1 shows the PRISMA checklist.
Studies were eligible for inclusion if they presented a prognostic model based on individual patient characteristics developed and/or validated in an upper-middle, lower-middle and low-income population as defined by the World Bank12 and presented data for adverse maternal or perinatal pregnancy outcomes. Examples of outcomes considered include gestational diabetes, hypertensive disorders of pregnancy, postpartum haemorrhage, maternal mortality, stillbirth, neonatal morbidity and mortality. The overview of LMICs according to the World Bank can be found in online supplementary data 2, a full list of outcomes considered and search strategy can be found in online supplementary data 3.
For this review, a prognostic model was defined as a model that could be used to estimate risks for individual patients or to distinguish groups of patients at different risk, based on ≥2 predictors.1 These models are often termed multivariable clinical risk prognostic models or risk scores. Prognostic studies were eligible if they reported on either (1) model derivation (ie, a new prognostic multivariable model was developed); (2) external validation (ie, an existing prognostic model was validated within an external cohort which had no connection to the cohort in which the model was developed) or (3) incremental value assessment (ie, a predictor was added or deleted from an existing prognostic model, which may result in a better prognostic value of the model).
Exclusion criteria were: case reports, reviews, letters, full text unavailable in English, wrong outcome (ie, not addressing adverse pregnancy outcomes), not developed or validated in LMICs.
Information sources and search strategy
The following electronic databases were searched: PubMed/Medline, EMBASE, Global Health Library and The Cochrane Library for publications. The search was conducted to include all publications from inception (1970) up to 2 May 2018. The search strategy was based on Geersing et al and terms related to LMICs, prognostic models and adverse pregnancy outcomes (see online supplementary data 3 for complete search strategy).13 No filters on language or other limits were applied. References of the previously published review by Kleinrouweler and included articles were reviewed for additional eligible articles (snowballing).
Duplicates were resolved and removed automatically and manually using EndNote. The web application Rayyan was used to screen articles on title and abstract.14 The screening of each article was performed blinded by at least two independent reviewers from a pool of four assessors (TH, JLB, GAK, MAC).
Disagreement among reviewers about the inclusion of a study was discussed until consensus was reached, including assessment of the full text of the article if necessary. Full texts of eligible articles were retrieved and assessed by one member of the review team (TH). If full-text articles were not available, one attempt was made to contact the first or corresponding author through ResearchGate.15
Data collection process and data items
Data extraction was performed by one author (TH) through a piloted standardised form, which was based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) checklist (online supplementary data 4).16 General study characteristics were recorded, for example, study design, inclusion criteria and setting. Methodical items were extracted regarding outcome, model development, type of validation and sample size. Model performance items were overall performance, calibration and discrimination measures. Finally, the final model or score chart was extracted when available.
For external validation articles, the general information of the model that was to be validated was extracted. Type of external validation, sample size, incidence of outcome and model performance were recorded. Data extraction for incremental value articles focused on the newly added predictors and model performance after adjustments.
Risk of bias in individual studies
The PROBAST (PRediction model Risk Of Bias ASsessment Tool) of risk assessment was used to define the risk of bias.17 The following themes were assessed: patient selection, predictors, outcomes and analysis. Within each theme different criteria were used to assess the quality of these studies. For example, within the theme about patient selection one of the criteria was ‘Were appropriate data sources used, for example, cohort, randomised controlled trial or nested case-control data?’ Or in the theme analysis ‘Was there a reasonable number of participants with the outcome?” Criteria could be answered with yes, no or unclear. Depending on the percentage of criteria that was correctly implemented in the methods of the different studies either a high (>75%), medium (50%–75%) or low (<50%) risk score was given per theme. The full risk of bias assessment table can be found in online supplementary data 4.
Synthesis of results
Results are presented narratively and in tables. No meta-analysis was attempted because none of the identified models examined the same definition/outcome.
Patient and public involvement
There was no patient or public involvement in this study.
Through the database search and snowballing a total of 1933 records were identified. After removing duplicates and excluding articles based on title/abstract and full-text screening a total of 20 studies remained, see figure 1, reporting on 21 prognostic models. Eighteen studies developed a model, two studies addressed incremental value and one performed an external validation (table 1).
Twenty-one prognostic models were identified for seven different outcomes in 20 articles (table 1): anaemia in pregnancy (n=1), gestational diabetes (n=2), hypertensive disorders of pregnancy (n=10), spontaneous preterm labour (n=1), cephalopelvic disproportion (n=1), postpartum haemorrhage (n=2) and neonatal death or stillbirth (n=4). The most common study design was the prospective cohort (n=11). Model performance was mostly judged by receiver operating curves, specificity, sensitivity, positive predictive value (PPV) and negative predictive value (NPV). The best performing model predicted spontaneous preterm labour and had an area under the curve (AUC) of 0.91 (sensitivity 88% and specificity 93%).18 Studies were conducted in 14 different countries, mostly in Brazil (n=7) and 4 studies were conducted in multiple LMICs.19–22 The development mostly occurred in (multiple) middle-income countries (n=23) and fewer in low-income countries (n=5). A complete overview of all reported study characteristics, for example, country, outcome, can be found in table 1. In eight studies, the complete final model was not presented, only predictors that were used in the derivation of the prognostic model but without their assigned weight. Studies were published between 1994 and 2018, most of them in 2009. Online supplementary data 5 presents an overview of all outcomes under investigation. Thirty different predictors were included in the identified prognostic models across all studies (table 2). They included maternal characteristics, current pregnancy-related symptoms, neonatal characteristics and additional laboratory investigations.
Of the 20 articles, 2 were incremental value assessments and 1 model was externally validated. The external validation was of the fullPIERS (Pre‐eclampsia Integrated Estimate of RiSk) model to predict adverse outcomes in pregnancies complicated by hypertensive disorders. The initial performance of the model was AUC of 0.88 (95% CI 0.84 to 0.92), and in the external LMIC cohort the performance was AUC of 0.77 (95% CI 0.72 to 0.82). This was considerably lower but still had good discriminative ability.22 The first incremental value article added serum biomarkers pregnancy-associated plasma protein A and placental growth factor to the pregnancy-induced hypertension prediction model of Antwi et al, and this improved model performance to AUC 0.95 (95% CI 0.87 to 1.00).23 The second article added blood oxygen saturation to the miniPIERS model and this resulted in an improved model performance, AUC of 0.79 (95% CI 0.75 to 0.85) and AUC of 0.81 (95% CI 0.76 to 0.86), respectively.21
A summary of the quality assessment is shown in table 3. The risk assessment results varied across studies and by assessed items. Overall, patient selection was generally considered low risk with 12 studies at low risk and 5 studies at medium risk of bias. High and medium risk was most often scored in the analysis quality assessment criteria, mainly because of a lack of reporting in the methods section, for example, on avoiding predictor selection based solely on univariate analysis, no reporting of internal validation was performed or how complexities in data (eg, censoring, competing risks and sampling of controls) were addressed. Some of the studies did not represent their final model equation in the article.
This systematic review identified 21 prognostic models developed and/or validated in LMICs for seven adverse pregnancy outcomes. Compared with the overall number of prognostic models in obstetrics as previously reported by Kleinrouweler et al (263 in 177 papers for 40 different outcomes; 8 of the LMIC models identified in this review were included), this accounts for <10% of available models globally.1 As such, the ‘10/90 gap’, the pattern that 10% of all global research funding is spent on the disease burden that afflicts 90% of the population in the world,24 seems to pertain into this field of study as well.
For global application of a prognostic model, predictors that are generalisable rather than context dependent are preferable—especially if they can be collected fast, easy, at point of care and low costs.25 For instance, maternal factors such as age, maternal weight and parity, blood pressure, Hb finger prick and patient-reported symptoms are preferred over more advanced techniques such as biomarker measurement or ultrasonography, which may not be readily available in low-resource settings. Similarly, although predictors related to a previous pregnancy’s history are generally available and a number of these well established as predictors of risk in subsequent pregnancies, these predictors were only used five times.26 27 Applicability of a model in LMIC could be considered during model derivation by selecting predictors that are appropriate for the setting in which the model will be implemented.9 We would like to encourage researchers when developing prognostic models to keep LMICs in mind and possibly provide a slimmed down model. In addition, implementation of models within a (low-resource) healthcare setting would require a presentation of the model that is user-friendly, for example, a score chart, nomogram or apps to increase uptake by healthcare providers. A number of included articles did consider this.28–30 The implementation of these models coupled with continued data collection on its performance allows for performance improvement of the models in practice and offers opportunities for validation of models in various settings.
Most studies included in this review developed new models. These require external validation before they can be used with confidence in clinical practice, as validation is a critical step to ensure that models perform similarly in new populations.31 Yet, we only found one external validation study conducted in a low-resource setting. The research lag in the development and validation of prognostic models in LMIC context also presents an opportunity as they allow to draw on the most recent methodological standards or through a focus on validating existing models in low-resource settings rather than (only) developing new models. In addition, within this process, incremental value assessment of specific predictors can be considered to improve performance in certain settings, or the derivation of ‘add on’ models with a basic set of predictors that can be expanded on with more advanced predictors depending on resources available.19 32
In this review, we focused on prognostic and not diagnostic models, that is, models that help to identify patients at risk of adverse outcomes and help caregivers in the triaging process to prevent adverse outcomes. Studies that developed models for diagnostic quality or care evaluation purposes, such as the effect of specific interventions on maternal health outcomes were not included. This includes ‘maternal-near-miss’ criteria studies as these are retrospective screening tools and obstetric diagnostic tools such as the ‘Edinburgh Postnatal Depression Scale’.33 34
To our knowledge, this is the first systematic review that provides a comprehensive overview of prognostic models for different outcomes in LMICs and assessed their quality using the PROBAST tool.17 A number of limitations were observed across the included articles that need to be taken into account in the interpretation of this review. First, several studies did not include their final prognostic model equation, making external validation by other researchers impossible and is problematic in judging the applicability of these models. Second, most articles did not assess their model’s performance as recommended in the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement35 with critical summary measures for a model’s performance such as the R2-brier score with a validation graph and the C-statistic with a receiver operating curve and after internal validation with bootstrapping. Instead, most studies included in this review reported only on sensitivity, specificity, PPV and NPV, which are measures that pertain to a single test (eg, when risks calculated by a prognostic model are dichotomised) rather than to overall model performance.36 37 Such measures can be informative, but only when presented alongside measures of overall model performance. Third, we excluded a number of articles based on language during title/abstract screening and at the full-text stage: Spanish (n=5), Portuguese (n=4), French (n=1) and Korean (n=1). Lastly, heterogeneity in outcome definition reduced comparability of performance across models, as in the definition of stillbirth after 20 or 32 weeks. In this respect, the initiatives to harmonise and generate consensus on outcome definitions as the CoRe Outcomes in Women’s and Newborn health (CROWN) initiative and the investment in reusability of data through the Findable, Accessible, Interoperable and Reusable (FAIR) guiding principles for scientific data management and stewardship are encouraging.38 39 A surprising finding was that none of the commonly used scores in obstetrics were identified, for example, Apgar, Bishop and Modified Early Obstetric Warning Score (MEOWS). We did not search specifically for these terms in our strategy but we assume that they should have appeared when searching for scor*. One study implemented the Apgar score as one of its predictors.40 This could point towards the tendency that well-established scores recommended by international guidelines and commonly used in clinical practice are not considered for external validation studies.
Despite the rapid development of prognostic models in obstetrics and frequent use since the Apgar score was introduced, only few implementation studies have been conducted to date, especially in low-resource settings. The development and validation of prognostic models needs to be ultimately coupled with an assessment of the impact in real-life healthcare settings potentially including randomised trials.41 Importantly, this evaluation should focus on the impact on tangible maternal and perinatal health outcomes and include cost-effectiveness, healthcare providers and pregnant women’s experiences, whether it can be implemented equitably and sustainably integrated and scaled up within an existing healthcare system.42 43
In conclusion, prognostic models can support healthcare providers in the delivery of antenatal, intrapartum and postpartum care services. Twenty-one different prognostic models identified were developed in LMICs. However, validation has hardly been conducted and is important before these models can be implemented with confidence in LIMC settings. Future high-income country prognostic model development should also pay attention to possible implementation in LMIC or provide simplified models that can be used in different settings.
Handling editor Sanni Yaya
Contributors TH and JLB: conceptualised the study and created the first version of the review protocol. GAK, MA-C, ES, BP, MR, KK-G, KB and DG critically reviewed the review protocol and approved it. TH, JB, GAK and MA-C screened eligible articles. TH extracted the data, supported by JB, ES and BP. TH drafted the first version of the manuscript, supported by JLB and BP. All authors contributed to data interpretation and critically assessed it. All authors gave approval for the final version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or the integrity of any part of the work are appropriately investigated and resolved.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.