New vector-control technologies to fight mosquito-borne diseases are urgently needed, the adoption of which depends on efficacy estimates from large-scale cluster-randomised trials (CRTs). The release of Wolbachia-infected mosquitoes is one promising strategy to curb dengue virus (DENV) transmission, and a recent CRT reported impressive reductions in dengue incidence following the release of these mosquitoes. Such trials can be affected by multiple sources of bias, however. We used mathematical models of DENV transmission during a CRT of Wolbachia-infected mosquitoes to explore three such biases: human movement, mosquito movement and coupled transmission dynamics between trial arms. We show that failure to account for each of these biases would lead to underestimated efficacy, and that the majority of this underestimation is due to a heretofore unrecognised bias caused by transmission coupling. Taken together, our findings suggest that Wolbachia-infected mosquitoes could be even more promising than the recent CRT suggested. By emphasising the importance of accounting for transmission coupling between arms, which requires a mathematical model, we highlight the key role that models can play in interpreting and extrapolating the results from trials of vector control interventions.
- Prevention strategies
- Cluster randomized trial
Data availability statement
Data are available in a public, open access repository. All code and other files to reproduce our results is available at: https://github.com/scavany/awed_trial_modeling/
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
A cluster-randomised trial has established that the release of Wolbachia-infected mosquitoes could be a highly effective intervention to reduce dengue virus transmission.
Cluster-randomised trials of interventions for infectious diseases can be subject to several biases when people or vectors move between areas covered by different arms of the trial, making the trial arms more similar.
We highlight three sources of bias that could be present in cluster-randomised trials of interventions against vectorborne diseases, the largest of which is due to not accounting for the coupling of transmission between trials arms.
The amount of bias introduced is likely to be greater for interventions of intermediate efficacy than those with high or low efficacy.
When designing and interpreting future trials it is important to properly account for transmission dynamics, and perhaps where needed to employ mathematical modellers in the analysis.
Dengue virus (DENV) poses a risk to around half the world’s population due to the widespread abundance of its Aedes mosquito vectors.1 Historically, the success of dengue control has been limited by challenges such as the expanding distribution of Aedes aegypti due to urbanisation and land-use changes, and ineffective or suboptimally applied control strategies.2 3 One novel control strategy that holds promise is the release of mosquitoes infected with Wolbachia, a vertically transmitted intracellular bacteria that reduces the ability of Aedes aegypti mosquitoes to transmit DENV.4 A cluster-randomised, controlled trial conducted between 2018 and 2020 in Yogyakarta, Indonesia (Applying Wolbachia to Eliminate Dengue, AWED)5 6 estimated that release of Wolbachia-infected mosquitoes had a protective efficacy against symptomatic, virologically confirmed dengue of 77.1% (95% CI 65.3% to 84.9%).7
Potential sources of bias in Wolbachia trials
There are at least three factors that can result in underestimated efficacy in this type of trial. All operate by making outcomes in treatment and control clusters appear more similar than if these factors were not at play, although they result in this for different reasons. First, the movement of humans between control and treated clusters means that individuals spend time outside of their allocated arm. This can reduce the time which study subjects residing in treatment clusters spend exposed to Wolbachia, as they will be spending more time in the control clusters, and vice versa.8 Second, the movement of mosquitoes between areas covered by different arms of the trial can lead to an appreciable proportion of mosquitoes in control clusters infected with Wolbachia, as Wolbachia-infected mosquitoes move from treatment to control clusters.7 This will lower these mosquitoes’ ability to transmit DENV and introduce a source of contamination across trial arms. This effect will occur much more strongly for the replacement strategy, when both male and female mosquitoes are released, and it is this strategy which we focus on here. Third, the dynamic, spatially localised nature of DENV transmission9 10 implies that suppression of transmission in treated clusters could influence transmission in neighbouring control clusters, thereby reducing incidence in both trial arms. While the first two effects directly affect individuals’ exposure to the intervention, this third effect describes the indirect effect due to DENV infection incidence depending on the prevalence in both arms. This occurs because reduced prevalence in the intervention arm results in less transmission and fewer introductions to the control arm, and vice versa. Hereafter, we refer to each of these three forms of bias as ‘human movement’, ‘mosquito movement’ and ‘transmission coupling’, respectively.
In their per-protocol analyses, Utarini et al7 acknowledged the potential effects of human and mosquito movement in their per-protocol analysis, and by incorporating recent travel and Wolbachia prevalence into their efficacy calculations did not detect a difference in efficacy from that estimated in the intention-to-treat analysis. Nevertheless, the analysis of the AWED trial by Utarini et al7 did not account for transmission coupling, and they noted that follow-up analyses were needed to further explore the potential for bias due to human and mosquito movement.
Understanding the magnitude of such biases is important when seeking to extrapolate the impact of interventions across contexts. Such extrapolation has been recently undetaken for the RTS,S/AS01 vaccine11 12 and the endectocide ivermectin13 for malaria. If failing to account for such transmission dynamics contributes to an underestimated biological effect of Wolbachia on DENV, we risk incorrectly assessing its broader impact. Given the myriad intervention options available to public health officials for dengue control,14 it is important for the potential impacts of each to be understood as well as possible.
Using mathematical models to understand sources of bias
In this analysis, we used a mathematical model of DENV transmission to gain insight into the possible magnitudes of the three aforementioned sources of bias. Our approach involved translating model inputs of the basic reproduction number (R0), the spatial scale of human movement (b), and the proportional reduction in R0 afforded by Wolbachia-infected mosquitoes (ε) into outputs of the infection attack rate (IAR) in control and treatment arms of a trial, in accordance with a seasonal, two-patch susceptible-infectious-recovered model.15 We used the outputs of IAR in treatment and control arms (IARt and IARc, respectively) to obtain an estimate of the OR of infection and, thereby, an estimate of the efficacy of the intervention, Eff=1 - OR. We constructed six different model versions for estimating efficacy, each of which includes different combinations of the three biases, all of them or none of them. Henceforth, we refer to the efficacy observed in the AWED trial as ‘observed efficacy’, and the efficacy estimated by a given model and ε as ‘estimated efficacy’. Finally, we quantify each bias as the difference in the efficacy estimated by a model including that bias and a model which does not include that bias (see online supplemental methods for more details).
We assumed a checkerboard pattern of control and treatment arms of 1 km2 to approximate the design used in the AWED trial, which covered the entire city of Yogyakarta, with neighbouring areas assigned to one arm or another in an (approximately) alternating pattern (figure 1A).7 We assume that individuals are evenly distributed within each cluster such that they have no internal spatial structure. The time that humans spend away from their home is assumed to follow a Laplace distribution (figure 1A, top right), which takes a single parameter, b, that we refer to as the scale of human movement. One way to conceptualise this would be that, at a randomly selected time in the day, the probability that an individual will be a given distance from home is given by the Laplace distribution. By assuming that individuals are evenly distributed within each cluster, we can then estimate the average proportion of time that individuals in each trial arm spend in their own arm (ρtt and ρcc) and in the opposite arm (ρtc and ρct—see the apportionment of time at risk section in Supplementary Methods for details). Larger values of b imply that people spend less time in their allocated arm, and for large values of b individuals spend roughly equal amounts of time in both arms (figure 1B).
How might the spatial scale of transmission and the dimensions of trial clusters affect efficacy estimates?
The relationship between the efficacy estimated by the model with all three forms of bias (the estimated efficacy) and the reduction in R0 (ε) was dependent on the amount of time people spent in their allocated arm (figure 1C)—the less time individuals spent in their allocated arm, the higher the reduction in R0 that was needed to recreate the observed efficacy from the AWED trial. If individuals spent less than 83.9% of their time in their allocated arm, it was impossible to generate the observed efficacy (77.1%), as that would have implied that ε exceeded 1. Assuming that individuals spent 92.9% of their time in their allocated arm (ie, ρii= 92.9%, corresponding to b=36.9 m—see the Spatial Scale of Human Movement section in Supplementary Methods for details and justification), we found that the observed efficacy (77.1% (95% CI: 65.3% to 84.9%)) corresponded to an ε of 49.9% (95% CI: 30.8% to 73.1%) (figure 1D, blue line). If we instead assumed that there was no movement between trial arms, we observed that much smaller values of ε were needed to explain the observed efficacy (6.3% (95% CI: 4.8% to 8.1%)). The difference between these estimates provides an indication of the extent of bias introduced by assuming that humans and mosquitoes remain in their allocated arms, when they in fact do not (figure 1D). We also explored the impact of assuming higher levels of movement between trial arms (b=58.4 m, the maximum amount of movement such that our model can reproduce the 95% CI of efficacy observed in the trial), finding qualitatively similar patterns (online supplemental fig. S11). In this case, the observed efficacy corresponded to an ε of 72.8% (95% CI: 47.2% to 100%).
When we fixed ε to the value that reproduces the observed efficacy in the AWED trial and increased human movement between arms by increasing b, the estimated efficacy by the model accounting for all three forms of bias decreased (figure 1E). For example, increasing the average distance in one direction between transmission pairs (b) from 36.9 m to 70 m caused a relative reduction of 20.0% in estimated efficacy, highlighting the sensitivity of efficacy to the spatial scale of human movement. This effect occurs for two reasons: first, as people spend less time in their allocated arm, the proportion of time that people spend under the intervention becomes more similar between arms; and second, in the presence of transmission coupling, a reduction in prevalence in the intervention arm reduces transmission in the control arm more as people spend less time in their allocated arm. Relatedly, estimated efficacy depended on the dimensions of the trial clusters, which we set to 1 km2 by default (figure 1F). When we reduced the cluster dimensions to 500 mx500 m, estimated efficacy dropped from 77.1% to 60.3%, representing a 21.8% relative reduction. This effect occurs because, as the cluster dimensions are reduced, people spend less time in their home cluster. Hence, the time spent in each trial arm approaches parity (ie, 50%). Increasing cluster dimensions above 1 km2 had somewhat less of an effect on estimated efficacy. For example, increasing the cluster dimensions to 2 km x 2 km resulted in an estimated efficacy of 86.7%, a relative increase of 12.4%.
What are the relative contributions of the different sources of bias in efficacy estimates?
Our approach enabled us to directly and separately model each of the three potential sources of bias: (1) mosquito movement, (2) human movement, and (3) transmission coupling. Movement of Wolbachia-infected mosquitoes is modelled by including a time-varying level of coverage, and we assume that mosquito movement does not contribute to DENV transmission (see Supplementary Methods—wMel coverage). When we assumed that ε was equal to 49.9%, allowing for mosquito movement but not human movement produced an estimated efficacy of 99.1%, because there was almost no transmission in the intervention arm in that case (figure 2A, online supplemental fig. S7). If we allowed for both mosquito movement and human movement, we observed a lower estimated efficacy of 93.6%. Although there was little transmission in the intervention arm in this case, individuals residing in the intervention arm could be infected in the control arm. Additionally, those assigned to the control arm experienced lower overall risk due to their time spent in the intervention arm. When we accounted for transmission coupling between trial arms alongside human and mosquito movement, thereby allowing for more transmission in the intervention arm, risk was the most similar across the trial arms of all scenarios, leading to the lowest estimated efficacy of 77.1% for an ε equal to 49.9%.
We quantified total bias as Eff(hmt) − Eff(0), where Eff(hmt) is the estimated efficacy under the model with all sources of bias and Eff(0) is the estimated efficacy under the model without human or mosquito movement. We then computed the difference in the bias produced by pairs of models to decompose overall bias into each of its three sources (figure 2B, online supplemental fig. S8-9, see Supplementary Methods for details). At the baseline ε of 49.9%, 17.6% of the total bias was attributable to mosquito movement, 8.3% to human movement, and 74.1% to transmission coupling. At all values of ε, the greatest source of bias was transmission coupling between trial arms. When ε was below a value of around 10%, the effective reproduction number at the start of the trial exceeded one in both arms. This value of ε varied slightly based on the model used (online supplemental fig. S8-9). If ε was below this critical value, increasing it in the context of coupled transmission reduced incidence in the control arm and caused smaller reductions in incidence in the intervention arm than if transmission had been uncoupled (online supplemental fig. S7, eg, panels D vs F). This implies that the bias introduced by transmission coupling increases as ε increases up to ~10% under our model’s parameterisation (figure 2B). Increasing ε past this point only leads to small reductions in incidence in the intervention arm in an uncoupled model, as incidence is already very low. Assuming more human movement between trial arms (b=58.4 m) led to qualitatively similar findings (figure 2C,D).
How can these biases be mitigated?
Biases arising due to human movement and mosquito movement are potentially addressable through careful statistical analysis of trial data or in the design of the trial.8 For instance, in the per-protocol analysis of the AWED trial, Utarini et al accounted for these two forms of bias by combining self-reported recent travel and local Wolbachia prevalence into an individual-level Wolbachia exposure index.7 Comparing groups with the highest and lowest Wolbachia exposure did not lead to higher efficacy estimates than their primary analysis. Another approach to addressing contamination involves describing the effectiveness of the intervention at the boundary between clusters using a sigmoid function.16–18 Our results suggest that failure to take steps such as this to account for human and mosquito movement would typically lead to underestimated efficacy, while failure to account for transmission coupling would lead to an even greater underestimate, particularly at intermediate reductions in R0.
Bias arising from human and mosquito movement could also be mitigated at the stage of planning the trial. The classical design to achieve this is the ‘fried egg’ design, in which the entirety of each cluster experiences their allocated conditions (ie, either intervention or control), but only the central part of each cluster is used for analysis.19 The principle behind this is that the samples selected for analysis are surrounded by a region receiving the same conditions, reducing the contamination between clusters, and more closely matching the conditions of full-scale implementation of an intervention. One drawback of the fried egg approach is that it assumes homogeneity within clusters, when in practice the central part of a cluster may be qualitatively different from the outer region—this likely limits the utility of this design for trials taking place in cities, such as the AWED trial. A more recently proposed approach involves excluding a subset of clusters from the trial completely, thereby increasing the distance between clusters and leading to disconnected clusters at less risk of contamination.20 This approach is based on the same principle as the fried egg design, but does not require homogeneity within clusters. While both of these approaches do mitigate the risk of contamination directly, they also necessitate a larger trial area and may be logistically infeasible in a trial taking place in a single city, as was the case for the AWED trial. Another approach could include reducing the number of clusters, but keeping the total area fixed, leading individuals to spend more time in their assigned arm and reducing mosquito movement by reducing the boundary between clusters. This approach also comes with disadvantages: first, fewer clusters is more likely to result in systematic differences between trials arms; and second, fewer clusters will lead to reduced replication in trials where the cluster is the unit of analysis. Our results show that the efficacy estimated from cluster-randomised, controlled trials of interventions against mosquito-borne diseases is highly sensitive to cluster size (figure 1F). Had the dimensions of the clusters in the AWED trial been much smaller, then the estimated efficacy may have been substantially lower. However, having fewer, yet larger clusters would likely introduce new biases by making the arms less comparable, which may not be an acceptable trade-off.
While bias due to human and mosquito movement can be mitigated through trial design and statistical methods, our results highlight a third source of bias, transmission coupling, that requires additional tools to fully address. Accounting for this bias first requires data on the spatial distribution of the intervention and on human movement. Data on human movement could be obtained from self-reported travel histories, as was done in the per-protocol analysis of the AWED trial, or from mobile phone data,21 for example. However, accounting for transmission coupling also requires interfacing these data with a dynamical transmission model to account for the fact that, in the presence of movement between arms, incidence in each arm depends on prevalence in both arms.22 Many common trial designs will lead to reduced bias due to transmission coupling—for instance, by allocating a greater proportion of the trial area to the control arm, with small intervention clusters situated among larger control clusters so that transmission suppression in the intervention arm has less of a population-level effect. The ratio of area allotted to treatment and control would depend on many factors, including the expected strength of the intervention, the local force of infection, and logistical constraints such as the size and length of the trial. Using a dynamical model synthesising these factors in the design of a trial could aid in understanding how different designs might affect bias due to transmission coupling.22 More work is needed to understand what types of spatial clustering patterns, among other features of trial design, would minimise this form of bias.
Limitations and caveats
Although our modelling approach allowed us to account for different potential sources of bias and to attribute the total bias to each of those sources, it has several caveats. First, our model was deterministic, yet stochasticity could be important for a highly efficacious intervention with potential to reduce transmission to very low levels.23 This simplification implies that our estimates are likely conservative, as these effects could increase the bias due to transmission coupling if a highly effective intervention increases the probability of transmission fadeouts. Second, our simple model does not reflect all of the complexities of DENV transmission. For example, we did not account for spatial heterogeneities in transmission or interactions between serotypes. Accurately quantifying the contribution of these effects to bias would require a more detailed model, but the qualitative results would likely be similar. Third, we did not calibrate our model to trial data, so incidence in our model may not reflect the actual incidence during the trial. However, our aim here was not to precisely quantify bias in the AWED trial, but rather to highlight some potential sources of bias in trials of that nature and to understand how these biases are influenced by transmission dynamics and human mobility. Moreover, our model was calibrated to actual incidence from past years in Yogyakarta, and so still reflects transmission typical of that location. It is also worth noting that an earlier version of our analysis, which used a simpler static model based on epidemic attack rate formulae, had qualitatively similar findings.24 Finally, we do not account for heterogeneity between clusters, such as regions of the city with systematically higher mosquito abundance, or greater human movement, or within clusters, such as that transmission may be higher at the edge of control clusters.
Without accounting for human movement, mosquito movement and transmission coupling, the efficacy of Wolbachia-infected mosquitoes as an intervention to control dengue is likely to be underestimated. As the estimate of efficacy in the AWED trial was already very high (77.1% (95% CI 65.3% to 84.9%))7 and, as we show, likely underestimated, Wolbachia-infected mosquitoes have potential to be a game-changing tool in the fight against dengue. Even as vaccines against dengue become available, a variety of vector control approaches are likely to remain key tools in the fight against dengue.2 14 Although we focused our analysis on a trial of Wolbachia-infected mosquitoes, our findings are applicable to any efficacy trial of an intervention that has the potential to contaminate the control arm, such as gene drive mosquitoes or ivermectin as interventions against malaria.25 26 As trials of these interventions continue, it will be important to learn what lessons we can from transmission dynamic modelling when designing and interpreting future trials to ensure that we understand the true promise of these interventions.
Data availability statement
Data are available in a public, open access repository. All code and other files to reproduce our results is available at: https://github.com/scavany/awed_trial_modeling/
Patient consent for publication
SC and JHH are joint first authors.
Handling editor Alberto L Garcia-Basteiro
SC and JHH contributed equally.
Contributors AP formulated the idea for the study, in conversation with other authors. SC, JHH and AP led the analysis and created most of the figures, with input from all authors. SC and JHH produced the first draft of the manuscript. All authors contributed to the final draft.
Funding This work was funded by the NIH National Institute of General Medical Sciences R35 MIRA programme (R35GM143029). John Huber was additionally supported by an NSF Graduate Research Fellowship.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.