Background

Calls to improve the rigour and reporting of evidence on the effectiveness of digital health programs, including mobile health (mHealth), are emerging.1 mHealth programs which aim to empower women and catalyse demand for health services through the provision of mobile health information have been shown to increase utilisation of antenatal care (ANC), skilled birth attendance and childhood immunisation rates in a number of settings.2–4 Less is known however about the performance of the messaging and technology platforms underpinning these programs; affecting exposure to program content and, in turn, the summative outcome measures observed. The failure to document and report on these processes and the linkages between message exposure and behaviour may hinder efforts to understand and attribute the effects observed to the program.

In Ghana, the Mobile Technology for Community Health (MOTECH)’s mobile midwife program sought to improve the uptake of reproductive, maternal, newborn and child health services by sending pregnant women and mothers of children under the age of 1 prerecorded audio health information messages timed to a women’s gestational age or her infant’s age. Limitations in the technological performance of the program5 meant that <25% of mobile health information messages were received by pregnant women.6 By 6–12 months post partum, <6% of enrolled women were exposed to at least one message.6 These findings reinforce the need to collect evidence on summative evaluation findings, and the processes which underpin them.

The MomConnect program was established in 2014 by the South African National Department of Health (NDoH) to register pregnancies, and provide pregnant and postpartum women with twice-weekly health information text messages as well as access to a helpdesk for patient queries and feedback.7 8 As one of only five maternal messaging programs to exceed 1 million registered users,9 MomConnect has grown to become one of the largest mHealth programs in the world. Beyond its absolute size, little is known about the factors influencing registration, population-level coverage or exposure to MomConnect’s health information content.

Expanding on efforts that describe the program,8 technology architecture10 and linkages between registration and adverse pregnancy outcomes,11 we explore evidence on the MomConnect program’s reach and messaging exposure. We follow the flow of data from individual and provider mobile devices on registration attempts, through to successful registration and message delivery. We differentiate individuals who have dropped out from those who actively opt to not receive messages through a short messaging service (SMS) request, after initial registration. Finally, we explore the effects of user characteristics on the timing of registration, opt-outs and drop-outs. This analysis aims to determine whether the program performs as intended and identifies opportunities for improving the program’s reach and messaging exposure.

What is MomConnect?

Details on the MomConnect program are presented elsewhere.12 13 In brief, MomConnect comprises two essential components: (1) maternal health information messaging and (2) a helpdesk. In this analysis, we focus on the former. Online supplementary table 1 summarises the message delivery sets for pregnant and postpartum women, while online supplementary table 2 summarises the message delivery content by thematic area and package. Maternal messages were developed by a consortium of stakeholders led by NDoH and inclusive of global maternal health content experts, academic partners, UN agencies, technology companies and non-government organisations. Fifty topics were identified to address relevant issues related to pregnancy or the newborn infant. For each topic, an SMS message was developed within the 160 characters allowed. Messages were kept simple and understandable to lay persons emphasising inspiration and action, and/or information and action with the broader aim of encouraging the mother to play an active role in the healthcare of herself and her infant. All messages were translated into South Africa’s 11 official languages and made available as an optional selection at registration.

Once finalised, maternal messages were bundled in two sets. The first set encourages pregnant women to attend facilities and receive ANC. The second focuses on essential newborn care, nutrition including infant feeding, immunisations and hygiene. Depending on the gestational age at the time of registration into MomConnect, women are eligible to receive one of three bundled message sets: (1) standard (sign-up prior to 30 weeks’ gestation); (2) later (sign-up at 31–35 weeks’ gestation) or (3) accelerated (sign-up at >35 weeks’ gestation).

Measuring exposure to maternal health information messages

Figure 1 outlines the optimal pathway for pregnant women from point of contact with the health system, to registration into MomConnect, message delivery and receipt and the intended effects on behaviour. In practice, multiple breaks can occur at each point along this pathway. Registration to receive MomConnect messages can be boosted through self-subscription or at community level via community health workers. Self-subscription or subscription from a community health worker at the community level triggers a short set of six messages mainly encouraging women to attend antenatal clinics. Eligibility to receive the full message set is dependent on facility-based confirmation of the pregnancy and facility-based registration via Unstructured Supplementary Service Data (USSD),ⁱⁱ either on the mothers’ mobile phone, on the clinic nurse’s mobile phone or captured in batches on a data clerks mobile phone. Registered women automatically receive messages. These continue until the baby is 1 year of age unless they otherwise ‘opt-out’ through SMS, either voluntarily or as a result of loss of the fetus or baby.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Measuring the flow of registration and message delivery. The red line denotes the pathway of registration flow while the light blue line reflects the message delivery pathway. The dotted nature of the lines is intended to denote the potential for breaks in the continuity of data flow at each point in the pathway. Analyses in this paper focus only on registration and message delivery data. WASP, Wireless Access Service Provider; Vumi/SEED, Messaging engine for the delivery of SMS; OpenHIM, Middleware system for enabling interoperability with health information systems; DHIS2, District Health Information System 2, includes data on antenatal care registration.

Table 1 summarises key definitions and measurement strategies. Exposure was measured by assessing the technological performance of the program as defined by: (1) USSD registration attempts; (2) USSD successful registrations and (3) the MomConnect platform’s effectiveness in ‘pushing’ out messages during pregnancy and post partum to registered women. Registration coverage was measured as the proportion of women successfully registered to MomConnect out of those reported in the District Health Information System 2 to have attended their first ANC clinic in a public health facility. Opt-outs are defined as registered MomConnect users who send an SMS declining future receipt of messages. Women were asked to specify one of the following reasons when opting out: miscarriage, stillbirth, baby loss, messages not useful and other. Distinct from opt-outs are registered users who ‘dropped-out’. Drop-outs were assessed at two time points: (1) unique phone numbers attempting to register to MomConnect which failed to successfully convert to registration and (2) registered users who failed to receive messages for five consecutive weeks. The latter were assumed to have changed their mobile phone numbers and/or lost their devices. Among registered users, message delivery success was determined as the proportion of messages received out of those sent.

All data used in this analysis were drawn from system generated data on registration, message delivery, opt-outs, drop-outs and ANC1 utilisation. Data on overall registration trends and ANC utilisation spanned from 1 January 2015 to 30 April 2017, while USSD data were assessed from 1 January to 31 December 2016. Data on USSD attempts, registration and message delivery trends were analysed using proportions and frequencies. Χ² tests were applied to assess the differences across provinces in user characteristics, including age, possession of a South Africa identification book or card (a proxy for nationality) and language. A logit model was then applied to registered users in each province to assess whether user characteristics were associated with the timing of registration, opt-outs and drop-outs. Clustering at the level of facilities was accounted for by the use of Huber White sandwich estimators. Adjusted ORs and 95% CIs are presented.

Registration into MomConnect

Figure 2 provides an overview of enrolment from 1 January 2016  to 31 December 2016. There were 426 631 unique phone numbers that attempted to register to MomConnect using USSD. An estimated 26% of these (n= 111 788) failed to convert to registration and were deemed to have dropped-out. Drop-outs during registration are attributed in part to challenges with the USSD platform which works in time-bound sessions and as well to human error in not completing the registration fields and/or dialing back in when session time-outs occur. For unique phone numbers successfully registered in 2016, a mean of 4.3 USSD sessions were required before registration was achieved. Among the 74% that did successfully register to MomConnect (n=314 843), 8% were identified as provider devices and 92% were women’s own mobile phones.ⁱⁱⁱ Collectively, this translates to 314 843 devices being used to register 532 030 women in 2016.^iv Provider devices (n=25 964) led to unique device (msisdn) registration of 243 151 women in 2016; a ratio of 9.4 registrations per device and accounted for 46% of total registrations. Over the same period, 288 879 (54%) women registered on their personal mobile device.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Overview of MomConnect enrolment flow from 1 January 2016 to 31 December 2016. USSD, Unstructured Supplementary Service Data .

Translating registrations into population level coverage

Table 2 presents data on the absolute number and characteristics of women successfully registered to MomConnect, while figure 3 presents trends in registration coverage overall and by province from 1 January 2015 to 30 April 2017. Nearly half of registrations came from two provinces: Gauteng (22%) and KwaZulu-Natal (KZN, 22%). In addition to having the lowest registration coverage as a percentage of ANC, the Northern Cape contributed 1% of MomConnect registrations (total population share 2%).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Percentage of MomConnect registered users out of those attending ANC1 from January 2015 to April 2017 by province.

National level trends suggest an increase over time in the proportion of women attending ANC1 registered to MomConnect from 40% in 2015, 55% in 2016, to 64% thus far in 2017. Coverage increased by a mean of 17% from 2015 to 2016 and 4% from 2016 to mid-2017. Across provinces, registration coverage was highest in Limpopo (89% in 2017) for all years. Slight declines in registration from 2016 to 2017 were observed in the Western Cape, North West, and Free State provinces.

We estimate that 83%–89%14 15 of ANC1 attendees own a mobile phone,^v a figure that if applied to 2016 data would translate to 805 268–863 481 women. Considering registration coverage as a percentage of ANC1 attendees with access to a mobile phone would shift coverage to 62%–66% in 2016 and 72%–77% in 2017. If registration attempts were converted to successful registrations, MomConnect’s coverage of ANC1 attendees would have been 74% in 2016 and 86% in 2017. When considered as percentage of ANC1 attendees with access to a mobile phone, addressing USSD conversion challenges could bring MomConnect registration coverage to 83%–89% in 2016 and 97%–100% in 2017.

In spite of USSD registration limitations, MomConnect is one of the largest maternal mobile messaging programs globally in terms of absolute numbers (>5 00 000 pregnant women in 2016) and with regard to the proportion of eligible women covered (>60% of all ANC1 attendees). Contextualising these data against other maternal messaging programs remains challenging because of the limited evidence available, particularly on registration coverage. The Mobile Alliance for Maternal Action (MAMA) project was one of the precursors to MomConnect in South Africa and elsewhere globally has deployments in Bangladesh, India and Nigeria.9 In South Africa, the MAMA project was implemented over a period of 3 years and led to the registration of over 500 000 women.16 In Bangladesh, MAMA is delivered through the Aponjon program, which provides maternal health information messages through IVR or SMS. Since its inception 6 years ago, over 1.9 million women have been subscribed and services expanded to include a doctor’s line and two additional mobile applications.9 In India, mMitra provides MAMA maternal messages using IVR to an estimated 600 000 subscribers across three states.9

Elsewhere, evidence on other maternal mobile messaging programs is emerging. The Kilkari program in India provides once weekly stage-based IVR health information messages on topics ranging from family planning to maternal and child nutrition to pregnant and postpartum women up to 1-year post partum. To date, the program has been scaled across 12 states and estimates having reached a minimum of 2 million pregnant women, new mothers and their families in the last 1.5 years; corresponding to a population level coverage of pregnant women of approximately 15%.17 In South Africa, the decision to enrol women at the facility level using mobile phones, while marred by challenges, has nevertheless meant that a larger proportion of pregnant women get covered and that nearly one-third of registrations occur within the first trimester of pregnancy thus maximising the period of exposure.

What are the characteristics of MomConnect registered users?

Significant differences were observed across provinces for all characteristics assessed (table 2). Findings suggest that the majority of registered users held a South African National ID card (58%–83%), were under 26 years of age (29%–42%), 13–26 weeks gestation at the time of registration (46%–54%) and opted to receive messages in English (28%–78%). In 6 of 9 provinces, over 25% of registered users did not submit information about a South African National ID card—a factor which may indicate significant utilisation of pregnancy services by foreign nationals. While half of all users are not registered until their second trimester, longitudinal trends in the mean timing of registration by gestational age suggest first trimester registrations have increased over time.

Table 3 summarises findings from a logit model exploring determinants of early registration from January 2015 to April 2017. Findings suggest that the odds of early registration were significantly higher among women between 26 and 35 years of age as compared with those <25 years of age or higher in most provinces. Individuals not in possession of a South African ID card had a slightly higher odds of early registration in KZN, Mpumalanga and North West provinces. Language was significantly associated with early registration in all provinces except the North West and Northern Cape.

Do registered users receive intended messages?

Figure 4 depicts trends in the proportion of messages successfully delivered by province over time, while online supplementary figure 1 presents trends in message delivery by mobile network operator (MNO). Users received over 80% of expected MomConnect messages over time. Delivery rates were stable across provinces but differed over time and by MNO type. By MNO, Telkom (71%) and Cell C (75%) reported the lowest message delivery success rates versus Vodacom (81%) and MTN (82%). Reasons for message delivery failure unfortunately were not available for all networks nor systematically assessed throughout the life of the program. However, based on the available data for 2016, the leading reason for non-delivery was that the SMS had expired—a likely indicator of an inactive phone number (online supplementary table 3).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Message delivery success rates by month and province from December 2016 to May 2017.

‘Churn’—defined by percentage of subscribers in a given time frame that cease to use mobile services for one reason or another—is an inevitable part of any program, particularly in settings such as South Africa where the majority of phones are prepaid. Given that MomConnect messages may be delivered for up to 21 months, from pregnancy to 1-year post partum, monitoring drop-outs due to churn is vital for optimising exposure to health information messages and avoiding unnecessary SMS costs. Unfortunately, MomConnect does not currently enable clients to change their registered phone number or follow-up in the event of message delivery failures. Future programs should optimally consider such measures and otherwise ensure that systems are in place to remove users in the event of repeated message delivery failures. In the case of MomConnect, in 2017 a system was put into place to classify registered users with five consecutive weeks of message delivery failure as ‘inactive’. Since this system has been in place, an estimated 1 28 839 users were discontinued—effectively saving the program costs which otherwise would have been lost in the system’s efforts to push out messages to target users no longer engaged.

This line of inquiry sought to emphasise the importance of understanding exposure to the program and likely variations in message delivery as function of the technology platform, MNO coverage and user engagement over time. Overall findings on message delivery trends are challenging to contextualise given the limited reporting on this in other mHealth programs. Unfortunately, few evaluations of digital health programs have adequately explored exposure and its probable linkages with health outcomes. In Zanzibar and Malawi, programs providing mobile health information messages to expectant mothers were associated with increases in the knowledge and utilisation of services across the continuum of care.2 18 19 While these findings are promising, the absence of details on individual level exposure to the program content greatly limits understanding of the factors influencing effects observed, their comparability with other mobile messaging programs, and generalisability to other settings. For MOTECH in Ghana, while outcome level data were not available, analyses of IVR message delivery trends suggested that 25% or less of expected mobile health information messages were received by pregnant women, despite the majority (>77%) owning a private mobile phone.6 For SMS programs like MomConnect, user engagement with the messages is not possible to measure through use of system generated data as the system only measures if the messages were delivered and cannot see if they were opened and/or read. However, even with slight variations by network provider, MomConnect’s overall message delivery success rate was high overall, by province, MNO and stable with program growth.

Do registered users stay engaged?

Registered users were able to discontinue messaging by either ‘opting-out’ or ‘dropping-out’ (table 1). Out of the 532 030 registrations in 2016, 10% (n=52 692) dropped out due to SMS delivery failures, 5% (n=24 857) opted-out and 85% went on to receive baby messages (figure 2). Individuals opting out were asked to specify the underlying reason from one of five prespecified categories (online supplementary figure 2). The leading reason for discontinuing messaging was ‘other’ (63%), followed by miscarriage (12%), stillbirth (10%), baby loss (7%) and messages not reportedly useful (7%).

Characteristics associated with opt-outs and drop-outs are presented in table 2, while table 4 presents data from a logit model exploring determinants of opt-outs and drop-outs among MomConnect users. Online supplementary table 3 compares the characteristics of registered users who opt-out and drop-out of MomConnect versus those that continue to receive messages. Registered users that opted to discontinue MomConnect messages (6%–9%) had similar characteristics to those that dropped out. The odds of opting out were significantly higher among individuals with a South African ID card, under 25 years of age and/or who did not speak English. By comparison, drop-outs were significantly higher among non-English speakers without a South African ID and under 25 years of age.