Impetus for the modelling resource

Following a relatively mild ‘first wave’ of COVID-19, which peaked in September, 2020, India witnessed an overwhelming onslaught of the Delta variant of the virus. This second wave, which reached its greatest height in early May 2021, had four times the peak reported cases as in the first wave.5 In a country as large and diverse as India, these events highlighted the need for strategic planning regarding resource allocation and infrastructure strengthening. As part of such preparedness, ICMR, the apex institute for biomedical investigation (research), led a collaboration with Imperial College London to develop a model of SARS-CoV-2 transmission in the country6–10 (see online supplemental figure S1, supporting information for an outline of the model). In June 2021, immediately following the decline of the second wave, the CHROMIC model was used to examine whether a third wave in India could be as severe as the second wave. In brief, results highlighted that such an outcome would occur if: (1) a new variant emerges that shows full immune escape from previously circulating variants (ie, against severe outcomes as well as infection), along with substantially increased transmissibility or (2) lockdowns in specific local areas showing high levels of transmission were suddenly relaxed. The subsequent emergence of the omicron wave validated the findings of this study: at its peak in January 2022, reported cases were two-thirds that of the peak during the second wave. While Omicron showed substantial immune escape allowing widespread reinfection, emerging evidence from other settings showed that prior immunity remained effective against severe disease, hospitalisation and death.11–13 Thus, the omicron wave was far milder than the preceding delta wave, in terms of demand for hospital-based care and mortality.

However, at the time of publication in June 2021 of the original model (which itself was developed in February 2021), none of this was apparent. The possibility of a severe third wave remained. While the published study focused on the national level, it was recognised that states and districts would benefit from the use of this model for their own planning, using locally relevant data. In particular, we aimed to develop a user-friendly, web-based ‘simulator’ through which state-level planners could interact with the model, with minimal training in modelling techniques (see figure 1 for an example screen from the simulator) and handful of input parameters.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Illustrative screenshot of the simulator. The left-hand panel includes all user-specified scenario specifications. The central panel shows model projections, including the first two waves, while the right-hand panel shows model projections for required hospital capacity in the third wave.

The aim of the simulator—developed through consultation with stakeholders, as described below—was to make projections for the hospital capacity that would be required in the event of a third wave. Given the uncertainty at the time on how a third wave might emerge, the simulator allowed users to specify different scenarios for mechanisms, including (1) the waning of immunity to previously circulating strains, (2) the transmissibility of any future variant, (3) the degree of immune escape of any such variant and (4) the release of local lockdowns and other restrictions in spite of emerging infections. The simulator also allowed users to specify scenarios for ramping up vaccination coverage, to simulate ways of mitigating the impact of a potential future wave. Importantly, the simulator-generated outputs for resource requirements drew heavily from the countrywide COVID-19 Registry14 and were therefore grounded in country-context and reality. The model underlying the simulator was written in Python, an open-source programming language. The simulator was developed through collaboration with Dure technologies, a locally present (India-based) consultancy firm (available here, https://covidwaveapp.duredemos.com). Overall, such three-way collaboration was unique in several respects; first, it involved close partnership between Government, academia and the private sector (Department of Health Research—ICMR, Imperial College and Dure technologies, respectively). Second, the model underlying the simulator was led by the scientists at ICMR, with support from Imperial College, ensuring that the ownership of the model and validation remains in the country in which it was to be deployed.

Participatory development of the simulator

Preparatory discussions: It was important to engage with relevant stakeholders from the beginning of development of the simulator, to ensure that they would have maximal input into its design and functionality. Early discussions involved senior policy-makers, health administrators and programme planners: we presented results of the published modelling work, and explained the aim of developing a user-friendly simulator based on this modelling (details in Supporting information), in order to help prepare for a potential third wave. We discussed the priority use cases of such a simulator; for example, whether it should focus on projections for symptomatic incidence, requirements for hospital capacity or the potential impact of increased vaccination, in mitigating health burden. Feedback highlighted that all three would be important, but that the highest priority should be placed on requirements for hospital capacity to mitigate impact while continuing aggressive vaccination. We also received requests for additional scenarios to be incorporated. For example, data from India suggested that not all of those admitted to hospital actually required oxygen support, suggesting that healthcare demand could possibly be better managed by encouraging patients to follow simple measures at community level before seeking hospital admission. We were requested to incorporate the potential impact of such measures, on demand for hospital-based care. Overall, we were advised to prioritise simplicity in the interface, by having a ‘basic and advanced’ input compartment as well as to minimise scientific jargon where possible.

Refinement of the simulator: Once an initial, functioning version of the interface was available, we re-engaged with stakeholders for feedback. Further, we piloted the simulator among a few state health officials. Box 1 shows some example feedback, illustrating the range of topics discussed. We were also advised to make the simulator more accessible for users who were not familiar with model parameters; principally, by providing guidance in pop-up ‘information bubbles’ on what would be considered ‘high’ or ‘low’ values for each of these parameters. One example is the value of for a third wave, reflecting the potential inherent transmissibility of a novel variant. We added help text to explain that ‘If the third wave arises due to a new variant, please enter the transmissibility of such variant as a percentage increase relative to Delta. A value of 0 (zero) means that the new variant is equally infectious to the Delta variant’. Similar, user-friendly guidance was provided for other third-wave parameters in the user’s control, including the rate at which immunity wanes, and the pace of ramping up vaccination coverage.

Another key topic covered in discussions was on the inherent risks of providing a modelling tool as a ‘black box’, in particular the potential for non-expert users to misinterpret model results. One important example was the role of uncertainty; the published version of the model incorporated uncertainty in model parameters, using Markov-Chain Monte-Carlo methods to propagate uncertainty from these inputs into uncertainty in model projections. However, it is important to note that this approach only partially addresses uncertainty. It does not address ‘structural’ uncertainties such as in the natural history of the infection, which would necessitate alternative model structures. Displaying parametric uncertainty in a user-friendly tool would therefore risk giving a false sense of confidence that the underlying model accounted for all sources of uncertainty. Instead, and as a result of these discussions, we adopted a simpler approach to uncertainty, encouraging users to use the simulator to explore different parameters, and to gain an understanding of which parameters matter most for third wave projections. Meanwhile, we would avoid showing parametric uncertainty in model projections, to avoid misinterpretation of these results.

Deployment and training: Once the simulator was completed, all state health authorities in India were encouraged by the union and state health secretaries through an official communication from the Ministry of Health and Family Welfare, Government of India, to use it to generate state-specific projections.15 Interested states were requested to contact ICMR for initial orientation-training on how to use and interpret the model. Participants for this training included potential users from the respective state and district health authorities. The participants mostly tended to be data scientists and epidemiologists, but also included senior public health officials who were interested in gaining familiarity with the simulator’s capabilities. Training, conducted online, began with a brief introduction to what mathematical modelling can and cannot do. It was emphasised that no model is fully predictive, and that models merely make projections on the basis of the best available data. Participants were then shown the simulator, and received a demonstration of how different third wave scenarios could be modelled. They were encouraged to try running the simulator on their own computers, in order to gain hands-on experience.

Uptake of the simulator: Overall, training was attended by over 700 officials from various districts in Indian states. Figure 2 shows outputs from website tracking, produced by Dure technologies, on the number of users visiting the simulator over time; there were 1612 visits to the website, with 1268 unique users as of 24 March 2022. At the central level, projections derived from the simulator helped to formulate travel advisories for known tourist destinations in India. These advisories were developed through a process of engagement of health mission officials of the relevant states.9 However, we caution that—owing to the rapid timeline over which the simulator was developed—it was not possible to instate formal mechanisms for monitoring how states used the simulator, in their decision making. We, therefore, do not have systematic data for the degree to which it influenced state planning, Moreover, and as noted above, the omicron wave in India was far milder, in terms of hospitalisations and deaths, than the delta wave. A more severe third wave would undoubtedly have seen even greater usage of the simulator, for health planning.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Timeline of the number of users of the CHROMIC Simulator for COVID-19.CHROMIC, Collaborative Health Research On Modelling-ICMR & Imperial College.

Recommendations

In the longer term, it is hoped that lessons learnt from this initiative could help to bolster pandemic preparedness in other countries in South and South-East Asia. By design, such preparedness would require enhancement of local capacity in development and use of modelling; critically, this would be focused not only on executing already-available models, but also on developing—and at the very least being able to critique—those models. The need for local district level data cannot be overemphasised as the onset of any wave, spread of infection and availability of health services can widely vary in different parts of a country as large and diverse as India. Thus, it is important for local evidence to be used to help locally relevant decision making. To this end, we offer the following, specific reflections and recommendations, arising from our experience during the COVID-19 pandemic.

Early preparation will be key in future preparedness, including collaborative network. The simulator was developed in a fast-changing pandemic situation, necessitating some compromises in its development. Given more time, we would have preferred to engage with a wider variety of possible users, including not only senior health officials, but also the state-level epidemiologists, data scientists and analysts who would be involved in usage of the tool. Even the simplest models can be complex to interpret correctly. Rather than providing the simulator for unsupported use en masse, in ideal circumstances we would have been able to establish direct rapport with users in each state, and provided follow-up training and guidance where needed. It is not feasible to establish these collaborative networks during emergency response to a pandemic. Instead, it will be important to incorporate modelling support as part of broader pandemic preparedness planning, establishing the necessary capacity and networks in advance of any future pandemic. Organisations such as the WHO, at country, regional and central levels, could play a key role in facilitating the establishment of these networks during ‘peace time’.

Design flexible modelling frameworks to respond rapidly to future emergencies: At a technical level, there are inherent challenges in developing valid models in advance of future pandemics. For example, modelling of COVID-19 was only possible once basic facts about its natural history were known.16–19 One possible approach to address this challenge is to pursue a modular approach, with modelling (‘back-end’) and simulator (‘front-end’) components readily being able to be updated independently. Under such a modular approach, a provisional ‘placeholder’ model—for example, a simple susceptible-exposed-infected-recovered framework—could be developed for the purpose of integration with the simulator. Countrywide engagement would then focus on the required outputs for the simulator (eg, required hospital capacity), as well as developing the network of users described above. Once a future pandemic emerges, how quickly the simulator can be deployed will depend only on how rapidly the model can be updated. Moreover, this model can be further refined as more data comes to light.

Communicating limitations of modelling: While user-friendly tools aim to lower barriers for participating in modelling, it is nonetheless critical for users to understand the limitations of modelling. We found that orientation-training was key for communicating important messages on what modelling can and cannot do, namely, users should be encouraged to actively enquire about the reasons that a given analysis yields the results that it does. Doing so will help immensely in understanding relationships between model assumptions and data on the one hand, and model outputs on the other. As noted above, model projections should never be construed as predictions, but rather as plausible projections based on available data. Furthermore, for any user-friendly modelling tool, there is a trade-off between model complexity and ease-of use. For the simulator, the underlying model is sufficiently straightforward for users to understand scientifically, as well as being computationally faster with the main purpose of planning and preparedness of the health services. Understanding these compromises were important for model transparency among users.

Establish mechanisms for monitoring policy uptake: Not all states would benefit from modelling, depending on their quality of data, existing capacity for implementing user-friendly tools, etc. Even states that would benefit from modelling may need support to maximise its use. To help identify such states, it is important to establish mechanisms for monitoring the use of any simulator in a future pandemic. Here, we have presented website monitoring statistics which, while helpful in informing usage, do not provide information on how the simulator was employed to support planning. In future, addressing this need will be an additional benefit of establishing the collaborative networks described above. As part of this network, users will be encouraged to report on how (if at all) the simulator has been used to inform decision making and, importantly, whether any additional functionality would enhance its value for such purpose.