Table 1

Questions to ask when designing and delivering virtual learning

PrincipleKey questionsTools/ guidance/case study, etc
Monitoring and Evaluation
1 Evaluate using complexity and realist approaches to create evidence of impact and good practice and support for implementation.In your project intervention design and monitoring plan have you considered: What works, for whom, in what respects, to what extent, in what contexts and how—and what change you aim to see in relation to this?Nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework built on a complexity assessment tool (CAT) methodology.24
Pedagogy
2 Coach for capabilities, facilitate vertical development and support innovation
  • Will your project prepare learners to apply their learning in different circumstances?

  • Will your project prepare learners to adapt how they apply their learning as circumstances change

  • What approach does your project take towards innovation development?

  • What motivates the learner and what are their key drivers?

  • How will your project draw on the existing knowledge and skills of the participant?

  • How has problem-based learning been integrated into the project?

  • How have you constructed the environment to foster relationships and multidirectional exchange between the Subject Matter Expert, Learner and Facilitator?

  • How have hard skills, attitudinal change, behavioural change and broadening of horizons been prioritised.

Understand any learning innovation as a complex intervention.25
Understand issues, such as adherence to guidelines.26
Understand the different approaches to innovation (eg,Transfer and Diffusion) but also the critiques of these approaches.27–29
The use of Learner Personas tools.30
3 Create connected and compassionate learning communities
  • How will your project bring people together to share and exchange learning?

  • What steps will you take to build relationships between the people involved in your initiative?

  • How will you ensure equity of participation?

  • What drives the relationships and exchanges between participants (networking opportunities, knowledge exchange, social connection, resource allocation)—how has the intervention been designed to use these drivers to foster relationship building?

  • What steps have been put in place to encourage exchange outside of the formal channels to build longevity of relationships and transition into the working world (eg, WhatsApp Groups, LinkedIn Groups)

  • Have you considered how to remove the practical frictions from distance/online group work?

  • Is synchronous time (Zoom Calls or webinars) being used for engagement and exchange or for knowledge transfer?

  • What tactics are being employed to appreciate the value of the personal experiences and insights that users share?

  • How has the intervention been structured to close the feeling of isolation that can be experienced with online or distance learning?

  • How is the individual made to feel part of the group and equally gain the feeling of being recognised by the group and the Subject Matter Expert?

Use participatory approaches.31 32
Include tools such as photovoice.33
Think through the ethical foundations of your work and consider taking a prioritarian approach.34 35
Compassion Rounds face.36
4 Develop coaching/facilitating faculty to support digital literacy and signpost to resources.
  • How will you strengthen local faculty?

  • Does your project support mentoring?

  • How will the learning be engaging for learners?

  • How will learning equip the learners to put their new knowledge into practice and how will this be evaluated?

  • Are the faculty equipped and trained in using online tools to engage with learners?

  • Are the faculty comfortable with the different pedagogical approaches best suited to online delivery

  • Has the intervention been designed to demonstrate what additional resources are required, for what and how their use will be evaluated?

  • What support structures have been put in place for the faculty?

  • For example, technical support resources, faculty meetings and review sessions, student feedback and evaluation process?

Understand approaches to supervision.37 Understand the concept of supportive supervision38 and the critiques of it.39
Do not rely on information dissemination models of learning. Understand how to use learning theory in developing your intervention40Understand the various approaches to understanding changes in practice, drawing on implementation science.41 42
Culture
5 Cocreate a person-centred approach focussed on inclusivity.
  • Has the learning been chosen by the learners?

  • Have you considered the full spectrum of stakeholders that should be included?

  • Have the learners shaped the nature of the learning?

  • Does the learning ensure a sense of belonging?

  • Does the learning help develop a sense of uniqueness

See below for inclusion framework and inclusion framework for induction.43
6 Build equity into the design through overt EDI and GESI approaches building mutuality and bidirectional learning.
  • Who has been involved in the design and delivery of the intervention?

  • Are different sectors of the community/society likely to benefit from the intervention?

  • Are there any barriers preventing the intervention from reaching everyone?

  • Have you considered the potential ‘equity stratifiers’ (ie, variables associated with differential access, uptake and/or outcomes)?

  • How will you monitor the effects of your project on EDI and GESI?

44–47
7 Develop culturally transferable adaptable capabilities.Does the intervention take into account the local context and what is relevant to local stakeholders?
How can the intervention be adapted to the local context and is there flexibility in the approach?
Have you unpacked what local context entails?For example, sex, disability, religion, social capital, socioeconomic status, local language, available infrastructure, sexual orientation, age, cost, geography, educational level, digital literacy, occupation, etc?
This is really important to get right. We need to make sure we have some flexibility built into the approach so interventions can be adapted.47
Organisational
8 Align with organisational priorities including ecological sustainability.
  • Does the intervention align with existing policies/ plans? If not, are there plans to adapt existing policies/plans?

  • Consider the different systems in which your intervention is situated, from the local to the international (Policies and plans do not always align with actual priorities, and there will be an interplay between the needs of both the organisation(s) and the individuals involved. Projects necessarily involve trading off some of these against others, and this is best done explicitly rather than tacitly. Trade-off analysis is a useful tool for this and fosters the necessary multistakeholder approach).

  • Consider: Who will use the system? here is the system? What affects the system?

48
9 Build a virtual learning organisation that improves quality.How will learning be identified throughout the project and used to improve the project? (Quality is often discussed and rarely defined. Building on the Trade-off analysis above, there are often different measures of quality which matter more to different stakeholders or are more relevant to one or other sub-system: for example, safety vs financial cost. An alternative framing is to consider stakeholder needs and explore how these can be met in a balanced way—for example patients need safe care but hospitals need to be able to afford it. This same focus on needs applies to learning: what is needed by the various stakeholders and how will they know define when their needs have been met?). Consider: What are the needs? How can the needs be met? How well are the needs met?WHO Toolkit48 49
10 Build an adaptive system.Can your project adapt as the context and priorities change? How will it do this? Consider: What is going on? What could go wrong? How can we make it better? (This builds again on the concepts of needs. These are dynamic and change as an individual changes and as their context changes. This requires an iterative approach which re-examines each aspect of the project recurrently. This could be embedded in a Kolb learning cycle, a plan-do-study-act (PDSA) cycle or a knowledge to action cycle.)48
  • EDI, equality, diversity and inclusion; GESI, gender equality and social inclusion.