You are here

  • Home
  • Archive
  • Volume 3, Issue 6
  • Using qualitative comparative analysis and theory of change to unravel the effects of a mental health intervention on service utilisation in Nepal

Article Text

Article menu

Download PDFPDF

XML
Research
Using qualitative comparative analysis and theory of change to unravel the effects of a mental health intervention on service utilisation in Nepal
  1. Erica Breuer1,
  2. Prasansa Subba2,
  3. Nagendra Luitel2,
  4. Mark Jordans3,
  5. Mary De Silva4,
  6. Bruno Marchal5,6,
  7. Crick Lund1,3
  1. 1 Alan J Flisher Centre for Public Mental Health, Department of Psychiatry and Mental Health, University of Cape Town, Cape Town, South Africa
  2. 2 TPO Nepal, Kathmandu, Nepal
  3. 3 Centre for Global Mental Health, Health Services and Population Research Department, Institute of Psychiatry, King’s College London, London, UK
  4. 4 Wellcome Trust, London, UK
  5. 5 Institute of Tropical Medicine, Antwerp, Belgium
  6. 6 School of Public Health, University of the Western Cape, Cape Town, South Africa
  1. Correspondence to Dr Erica Breuer; erica.breuer{at}uct.ac.za

Abstract

Background The integration of mental health services into primary care is essential to improve the coverage of mental health services in low resource settings, but the evaluation of this remains challenging. We used a programme’s Theory of Change (ToC) as a conceptual framework to determine what combination(s) of conditions at facility and community level influenced the mental health service utilisation as a result of a district mental healthcare plan (MHCP) implemented in Chitwan, Nepal. In addition, we show how qualitative comparative analysis can be used to provide an integrated analysis of data from a ToC.

Methods We conducted a longitudinal case study of 10 health facilities where the MHCP was implemented. We collected data from all facilities at baseline (October to December 2013) and quarterly following the implementation of the intervention (March 2014 to November 2016). The data were analysed using pooled qualitative comparative analysis in fsQCA V.2.5.

Results The following conditions were necessary for high mental health service utilisation: presence of basic and advanced psychosocial care, evidence-based identification and treatment guidelines (WHO mhGAP), referral to tertiary services and the presence of trained female community health volunteers . Two additional combinations of conditions were also identified as sufficient for a high mental health service utilisation: high medication supply, trained facility staff and either the use of a community informant detection tool or having a larger proportion of the community attend community awareness activities.

Conclusions Both supply-side interventions (formalised approaches to health worker detection and treatment, training of health workers, supervision) and demand-side interventions (community awareness and case finding) are important to integrate mental health in primary care. ToC can be used to provide an integrated analysis of data from a ToC, therefore helping to shed light on the black box of complex multilevel interventions.

  • theory of change
  • qualitative comparative analysis
  • routine longitudinal data
  • programme theory
  • mental health

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjgh-2018-001023

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Key questions

    What is already known?

    • Multiple intervention components, such as training, supervision and increasing medication supply, are required to integrate mental health into primary healthcare in low resource settings.

    • The relative importance of these in contributing to an increase in mental health service utilisation is not known.

    What are the new findings?

    • The study showed that mental health service utilisation was high when a combination of supply-side and demand-side intervention components worked together.

    What do the new findings imply?

    • Both supply-side and demand-side intervention components are necessary to improve mental health service utilization.

    Background

    There is increasing momentum for the integration of mental health services into primary healthcare to address the mental health treatment gap, which is especially marked in low-income and middle-income countries.1 Barriers to integration such as including low investment in mental health, staff resistance to decentralisation, few people trained in mental healthcare and overburdened primary healthcare staff with minimal supervision2 remain.

    Integration of mental health services into primary healthcare requires multiple intervention components, including case finding, assessment, and psychosocial and pharmacological treatment.3 4 The role of mental health specialists shifts towards providing supervision for primary care staff, consultation for complex cases and assessment and treatment of cases that cannot be managed at primary care level.3 Despite the evidence for the each of the above components of care in the literature5–8 and the development of evidence-based guidelines in the form of mhGAP to support this integration by WHO,9 there is little guidance for the implementation and effectiveness of these intervention components implemented into routine care in low resource settings.5 9 Developing such guidance is difficult because evaluation of the implementation of multiple complex health interventions into routine health services is challenging, particularly when the implementation relies on health system resources such as staff, medication and space,10 and is constrained by what is feasible within the health system.11

    As part of a multiyear mental health services implementation research project, the Programme for Improving Mental Healthcare (PRIME),12 Jordans and colleagues13 developed and implemented a district mental healthcare plan (MHCP) in West Chitwan, Nepal, which focused on the detection and treatment of depression, alcohol use disorders, epilepsy and psychosis. It was developed through comprehensive formative work,14 including the development of a Theory of Change (ToC), and pilot testing in two health facilities13 and is described in detail below. Although most of the intervention components were evidence based, it is not clear which components of the MHCP are essential in what combination for an increase in mental health service utilisation at facility level in Nepal.

    Conceptual framework for the evaluation

    We developed a ToC to assist with both the development and evaluation of MHCPs15 in the five PRIME countries, including Nepal.13 ToC is a theory-driven planning and evaluation approach that identifies the short-term, medium-term and long-term outcomes of the intervention in order to unpack the ‘black box’ of complex interventions.16 We use the Medical Research Council Guidance definition of complex interventions: interventions which contain multiple interventions, causal strands, outcomes, groups and organisational levels.11 The PRIME ToC outlined the programme theory for how the intervention would lead to improved health, social and economic outcomes for people living with priority disorders and their families and carers in the district.15 17 The PRIME ToC outlines the outcomes required to achieve this impact in relation to political buy-in, capacity building, identification of mental illness and treatment and care at district, facility and community level (online supplementary file 1). Mapped onto these were the intervention components required to achieve these short-term and medium-term outcomes, as well as indicators to measure progress. This process and the resultant ToC is described in detail elsewhere.15 17 A summary ToC is shown in figure 1 with the detailed ToC included in online supplementary file 1.

    Supplemental material

    Figure 1
    Figure 1

    PRIME Summary Theory of Change at district, facility and community levels adapted from Breuer et al.15 17

    Although the ToC provided us with a framework to develop the PRIME MHCPs, results from a recent systematic review found no papers that evaluated process and outcomes simultaneously.18 Qualitative comparative analysis (QCA), a case-oriented method developed in the social sciences for analysis of data in comparative case studies, holds promise as a way to do this.19

    In summary, it is important to understand which mental health intervention components result in more people accessing services leading to improved health, social and economic outcomes. However, evaluating complex mental health interventions integrated into routine settings is difficult, particularly when using routine data. The PRIME ToC provides a conceptual framework for the evaluation of these interventions although there is little guidance on how the process and outcomes data from ToC can be combined in one analysis. This paper aims to determine what combination(s) of conditions identified by the PRIME ToC at facility and community level influenced the mental health service utilisation in the PRIME implementation facilities in West Chitwan. In addition, we show how QCA can be used to provide an integrated analysis of data from a ToC.

    Methods

    We conducted a longitudinal case study based on indicators developed specifically to measure the PRIME ToC indicators,15 20 including the collection of routine service utilisation data.

    Setting and intervention

    The mental healthcare situation in Nepal exemplifies many of the global challenges to the integration of mental healthcare. Nepal has a recent history of insurgency followed by a delay in the drafting of the constitution.21 Although Nepal’s mental health policy was drafted in 1997 and includes the provision of mental health services in primary healthcare, it has not been implemented.22

    This study was set in Chitwan (population 579 984),23 the implementation area for the Programme for Improving Mental Healthcare in southern Nepal. The MHCP was implemented in 10 health facilities and their surrounding areas. Nine of these were health posts which provided outpatient services and one was a primary healthcare centre, which also provides emergency services. Female community health volunteers (FCHVs) who are trained in basic health promotion and referral are based in the community surrounding the health facilities.

    Prior to the implementation of PRIME in Chitwan, there had been ad hoc mental health training by NGOs and tertiary centres for primary healthcare staff. Lack of supervision and refresher training meant that in practice, there was no consistent mental health service provided in primary care.21 In addition, there was a limited and erratic supply of psychotropic medications (phenobarbitone and amitriptyline only), no counselling or psychotherapeutic services available and no standardised guidelines or treatment protocols.21

    The PRIME MHCP was implemented at the district, facility and community levels. At the district level, engagement and advocacy activities were conducted with health service managers, and ensuring there was a functioning referral system for complex or treatment-resistant cases to specialist mental health services. At the health facility level, the plan included service provider awareness, screening and assessment following the WHO mhGAP intervention guide,24 brief psychosocial support derived from structured interventions adapted from the Healthy Activity Programme (HAP)25 and Counselling for Alcohol Problems (CAP),26 pharmacological management and follow-up for monitoring and treatment adherence. At the community level, the MHCP included community awareness programmes conducted by PRIME community counsellors in coordination with the FCHVs, proactive case detection in the community using the community informant detection tool (CIDT)27 as well as the provision of the HAP and CAP by a community counsellor. The CIDT is a novel vignette-based case finding tool, which was developed for use in Nepal.27 28

    Sample and measures

    All of the 10 health facilities in Chitwan where the PRIME MHCPs were implemented at each time point were selected as cases. The health facilities are described in table 1.

    View this table:
    Table 1

    PRIME mental healthcare plan implementation facilities in Chitwan, Nepal

    The majority of the data used in this study were collected through a quarterly profile of the implementation facilities designed to collect facility-level indicators outlined in the PRIME ToC20 and table 2. A baseline profile was conducted to collect data from October to December 2013. Following implementation of the district mental healthcare plans (beginning in March 2014), the profiles were collected quarterly until November 2016. Data collected included monthly health information system data on mental health patients (which were averaged over the quarter) and quarterly data on trained staff, supervision, medication supply and presence of PRIME intervention components. Data at the community level were collected by the programme implementers.

    View this table:
    Table 2

    Outcome and condition sets, indicators and calibration for qualitative comparative analysis

    Data analysis

    We analysed the data using QCA, a case-oriented method developed in the social sciences for analysis of data in comparative case studies.29 It can be used to analyse both quantitative and qualitative data from multiple sources by using set theory and Boolean algebra to identify patterns in data. QCA identifies both necessary conditions, that is, those that are necessary for the outcome to occur but not sufficient, and sufficient conditions, that is, those that lead to the outcomes in some cases.30 Rather than identify single conditions or variables which lead to an outcome, QCA seeks to identify causal pathways or configurations of causes in diverse contexts.31 The relative importance conditions can be determined by (1) whether they are identified as necessary conditions with higher consistency scores indicating a stronger relationship with the outcome and (2) conditions that are included in more than one causal pathway of sufficient causes.

    QCA is particularly suited for small numbers of cases where statistical analysis is not possible. Inherent in QCA is equifinality, that is, that there can be multiple causal factors within a pathway or across distinct pathways to an outcome.31 This recognises causal complexity,32 unlike many statistical methods which isolate the effects of one factor on the outcome by controlling for all other factors.

    QCA is asymmetrical. This means that the causal pathways shown to lead to a lack of outcome are not merely the inverse of those leading to the outcome but could be completely different causal pathways. For more detail on the QCA approach, we refer to Kane et al.19

    QCA outcomes and condition sets

    We used the PRIME Summary ToC (figure 1) as our conceptual framework to determine the hypothesised facility-level and community-level pathways which led to increased service utilisation, that is, the ToC outcome: “People with mental disorders are identified and/or diagnosed in the facility”.

    We used the section of the ToC leading up to the service utilisation to identify the condition sets, which are the conditions of interest which may or may not result in the outcome. These are listed in the second and third columns of table 2. We included all indicators for the short-term and medium-term outcomes in the subsection of the PRIME ToC as condition sets except for three: (1) programme approved and budget available at district level; (2) programme co-ordinator in post; (3) people with mental disorders are willing to seek treatment. The first two were excluded because the data were collected at district level and therefore did not vary across facilities and therefore could not be included in the QCA. Therefore, we have limited this analysis to understanding the community-level and facility-level pathways outlined in the ToC (as indicated in the grey boxes in figure 1).

    The third was excluded because these data were collected qualitatively and the sample size at each of the facilities was not large enough to be disaggregated by facility.

    We added two additional condition sets not in the ToC: (1) availability of mhGAP protocols for detection and treatment of mental illness and (2) referral to tertiary care. The first was added to mirror the existing indicator, availability of psychosocial interventions, and the second had been included further along the ToC. However, the availability of referral mechanisms could also contribute to willingness of healthcare workers to identify people with mental illness. Therefore, we included it as a condition set in this analysis.

    Data preparation and calibration

    In order to minimise the number of condition sets, multiple indicators for one condition were aggregated. For example, we averaged medication supply for all medications outlined in the MHCP to create a summary condition for medication supply. Where we were unsure whether to combine the indicators from a conceptual point of view, we conducted a factor analysis to ensure they loaded on the same factor. For example, supervision for pharmacological and psychological management at facility level were combined into one condition. The ToC indicators and the process used to combine indicators are outlined in table 2, column 3.

    Once the outcome and condition sets were finalised, each case was calibrated by assigning a value between 0 and 1 under each condition set to describe set membership or the degree to which the case meets that condition. Zero is fully out of the set, 1 fully in the set and 0.5 neither in nor out of the set.

    The data for the outcome and each condition set were calibrated in various ways, using a theoretical understanding of the indicators and calibration commands in the fuzzy package in STATA.33 34 To aid the interpretation of the results, we ensured that the data were calibrated carefully to ensure meaningful interpretation for programmatic and policy outcomes. The outcome, for example, was calibrated as follows: monthly mental health patients as a proportion of all monthly facility patients was averaged over each quarter. The range across the implementation period was 0%–10.69% with a median of 1.79%. These proportions were graphed over time to determine the pattern at baseline, immediately following implementation and over the remaining period of implementation. In most facilities, the percentage of patients attending for mental illness increased to above 2% during the implementation, with 25% reaching 3% and above. We therefore decided that 0% was fully out of the set, 1% neither in nor out of the set and 3% fully in the set. We used a direct transformation in STATA to calibrate levels of deviation from these anchors in terms of log odds and to obtain the fuzzy values.35 The calibration approaches for the conditions are outlined in table 2. The calibrated data can be found in online supplementary file 2.

    Supplemental material

    Data analysis

    The data were analysed using pooled QCA36 in fsQCA V.2.537 where data are pooled for all facilities and outcomes and which therefore treats each observation for each facility and each time point as an independent observation.

    We conducted each of the following analysis steps for the outcomes ‘high’ and ‘low mental health service utilisation’ as QCA is asymmetrical. We first identified necessary conditions from the list of conditions outlined in table 2 using the Necessary Conditions option in fsQCA and XY plots. Necessary conditions are conditions for which the outcome is almost always a result of a causal condition.19 As outlined by Garson, we defined conditions as necessary when the consistency, or degree to which the outcome was a result of the condition, was greater than 0.90.30

    Second, we constructed a truth table with the remaining conditions to determine sufficient causal pathways. A truth table is an intermediate step in the QCA analysis that shows all possible configurations of conditions, and the number of cases that exhibit each causal configuration, together with a consistency score for the causal configuration. The abbreviated truth tables containing only the causal configurations for which data were available for the outcomes of high and low mental health service utilisation are presented in online supplementary file 3. We applied a frequency threshold of 2 and used a standard consistency threshold of 0.830 and the standard analysis option in fsQCA.

    Supplemental material

    fsQCA uses Boolean miniminisation to simplify the combinations of conditions and provide sufficient combinations of conditions associated with the outcome. fsQCA provides three types of solution according to the simplifying assumptions used: (1) a complex solution, which minimises the solution based on the available data only; (2) an intermediate solution, which uses theoretical knowledge to determine which logical remainders to include; and (3) a parsimonious solution, which finds the simplest solution without evaluating the plausibility.38 For the intermediate solution, logical remainders were coded according to the programme theory outlined in the ToC. Logical remainders are the causal configurations in the truth table for which there were no available data.

    We calculated solution consistency and coverage based on the findings. Coverage refers to the degree to which the causal condition explains the outcome in the sample.19 There is no specific cut-off for coverage because a lower coverage may indicate a more uncommon causal pathway. We present both the intermediate and complex solutions of sufficient conditions from the truth table analysis in the table and only the intermediate solutions in the narrative.

    Permission to collect data was obtained from the managers of all 10 facilities to obtain facility-level data.

    Results

    The results of the QCA for high and low mental health service utilisation at primary care facilities are outlined in tables 3 and 4, respectively.

    View this table:
    Table 3

    Necessary and sufficient causes for high mental health service utilisation

    View this table:
    Table 4

    Sufficient causes for low mental health service utilisation

    In summary, the presence of basic and advanced psychosocial care (consistency 0.97, coverage 0.70), mhGAP (consistency 0.92, coverage 0.77), referral to tertiary services (consistency 1.00, coverage 0.63) and trained FCHVs (consistency 0.97, coverage 0.74) were necessary but not sufficient for the outcome of high mental health service utilisation.

    In addition, two causal pathways were identified as sufficient for a high mental health service utilisation in the intermediate solution of the truth table analysis. Mental health service utilisation was high in facilities at times when medication supply was high, facility staff were trained and either the CIDT was used frequently or a larger proportion of the community had attended community awareness activities. The solution consistency was high at 0.85 and with a moderate coverage of 0.74. This indicates that these combinations of conditions lead to the outcome in a high proportion of cases and that a moderate proportion of the cases with the outcome are a result of this combination of conditions.

    There were no necessary conditions for low mental health service utilisation at primary care facilities and one causal pathway was identified as sufficient in the intermediate solution in the truth table analysis. Mental health case load was low in facilities at times when there was lack of facility and community supervision, lack of mhGAP services and a low proportion of the community members attending community awareness activities. The consistency was high for this solution (0.90), but the coverage moderate (0.56). This indicates that these combinations of conditions lead to the outcome in a high proportion of cases, but that only a moderate proportion of the cases with low service utilisation are the result of this combination of conditions.

    Discussion

    This study shows that in Nepal, a combination of formal mental health service intervention components, training, medication and community intervention components lead to increased mental health service utilisation in primary care facilities. In contrast, at times when mhGAP availability, supervision and community awareness activities were low, mental health service utilisation was also low. This indicates that different inputs or intervention components may be important to the establishment and sustainability of mental health services in the community.

    The identification of three defined treatment packages (mhGAP, psychosocial services, referral) as necessary conditions over and above inputs such as training and medication for an increase in service utilisation is in line with the existing evidence on core packages of care.3–5 These services were defined as follows: mhGAP services were available when all the required inputs were in place. Specifically, the appropriate staff were trained and medication, mental health outpatient register, protocols, mhGAP flowchart and brochures were available. Similarly, basic and advanced psychosocial services were defined as available when the appropriate staff were trained and the mental health outpatient register, basic psychosocial manual, HAP/CAP manual, brochures and flowchart were available. Our results show that formal guidelines and reporting and referral need to be in place for an increase in detection and treatment by primary healthcare workers. This can be seen in the increase in service utilisation, which occurred following the formalisation of these services in the clinic rather than when training alone was conducted. The results of the truth table analysis also indicated that training needed to be combined with medication and either community awareness activities or case finding in the community. This may give some insight into why the evidence for the effectiveness of training alone on detection of mental illness is mixed, particularly in the long term.39–42

    This study highlights the importance of creating demand at community level. Trained FCHVs were a necessary condition of high mental health service utilisation. In the MHCP, FCHVs are responsible for a large proportion of demand creation activities, namely administering the community detection tool (CIDT) and organising community awareness programmes which are conducted by the community counsellors.13 FCHVs also provide home-based care services for patients who have stopped attending the facility. This includes psychoeducation, emotional support and motivation to seek services at the health facility. For high mental health service utilisation, the two sufficient causal pathways included both facility-level conditions (training and medication provision) and either high use of the CIDT or a higher proportion of the community reached with community awareness programmes. The findings in this study of the importance of demand-side factors corroborate the findings of a qualitative study in Chitwan exploring demand and access conducted in the intervention development phase of PRIME.43 Brenman and colleagues43 used framework analysis of qualitative interviews and focus group discussions with stakeholders with the PRIME ToC as their conceptual framework. They found that many of the barriers to demand and access to mental health services were at the community level, including high levels of stigma around mental health, low mental health awareness and lack of awareness about treatment availability. They also suggested that in addition to service provision at the facility level, working at the community level to increase awareness of mental illness and working with local understandings of mental illness was essential to ensure uptake of services. A community-based survey of barriers to care in Chitwan showed that stigmatising beliefs in the community were a barrier to seeking care.44

    We found that supervision was important in ensuring that detection and treatment for mental illness occurs at primary healthcare. Low mental health service utilisation occurred at facilities and times when supervision at facility and community level was low, mhGAP was unavailable and low numbers of people in the community were reached by community awareness programmes. The majority of data contributing to this causal pathway is from the initial roll-out of the MHCP when training had been conducted but supervision, mhGAP and community awareness programmes had not reached a substantial amount of people. Understanding the role of lack of supervision and low community awareness and lack of formal mhGAP services is important because integration of mental healthcare into primary care often consists of short training courses and very little supervision at the district level.45 This is due to many factors, including the lack of trained mental health specialists to offer supervision and the lack of evidence for the type and quality of supervision required.46 However, this study shows that supervision at both the community and facility level together may be important to avoid low service utilisation. Structured and ongoing supervision should therefore be seen as a key component of a successful mental healthcare plan together with facility-level and community-level components.

    The main contribution of this study is to provide evidence to support a part of the PRIME ToC, which identified specific short-term and medium-term outcomes that work together at facility and community level to improve identification of people with mental illness in facilities in southern Nepal. In this analysis, we showed that all short-term and medium-term outcomes included from the PRIME ToC were either necessary or sufficient (in combination with other conditions) in cases of high mental health service utilisation or absent in cases of low service utilisation. This study did not identify specific gaps in the ToC, although the lower coverage of the intermediate solution for the lack of outcome could indicate that there are other causal pathways that have not been identified in this analysis. These could include, for example, the level of facility manager buy-in to the provision of mental health services, quality of facility management, quality of supervision, competing priorities within the health service or contextual factors.

    This study makes a methodological contribution as an example of how QCA can be used together with a ToC to analyse routine service implementation data. There have been some recent applications of QCA to health services research.31 32 47 48 However, to our knowledge, there have been no studies which have used QCA and ToC or QCA for longitudinal data analysis in health services research. ToC provides a transparent approach to develop a contextually relevant conceptual framework and indicators which can be used to quantify the QCA condition sets and outcomes. QCA can then be used to evaluate where and how the different factors identified by the ToC affect the outcome. QCA is particularly suitable for the analysis of routine implementation data because it seeks to understand causal complexity rather than identify individual risk factors,19 uses a relatively small number of cases (in this case health facilities) and can combine data from different sources, including qualitative data. We propose that the combination of ToC and QCA could be expanded beyond health services data to combine process and outcome evaluations for randomised controlled trials. For example, where mixed-method data were available for trial participants, a QCA could be used to understand how fidelity, dose, participant characteristics, therapist characteristics and other contextual factors are related to the outcome.

    During the process of applying QCA together with ToC, we learnt several lessons. First, using a conceptual framework, such as ToC, to guide QCA is important to identify the outcome and condition sets as well as providing a theoretical basis for how to treat logical remainders. Second, the number of conditions needs to be kept to around 5–7 to avoid the ‘limited diversity problem’19 which occurs when there are too many conditions in relation to the number of cases which prevent Boolean minimisation of the truth tables and result in causal pathways with many conditions and low coverage. This is similar to the problem in frequentist statistics of including too many explanatory variables in a multivariate linear regression without an adequate sample size. To avoid this problem, we aggregated similar indicators prior to data analysis, for example, proportion of trained prescribers and non-prescribers and the number of trained home-based care workers and FCHVs. Other approaches to avoid limited diversity include (1) a comprehensive approach where all conditions are included iteratively in the analysis until the most parsimonious solution is reached or (2) a two-step approach where an analysis of ths distal conditions is conducted first, followed by the proximal conditions.49 Third, data calibration is one of the key steps in QCA and should be conducted with a theoretical understanding of the values chosen, that is, that 1 is fully in the set, 0 is fully out of the set and that 0.5 is neither in nor out of the set. The calibration of data should be conducted using thresholds that are meaningful to the programme. Although tools exist in STATA and other software to calibrate continuous variables, the distribution of the resulting calibrated data should be checked to ensure that the interpretation of the data is in line with the theoretical understanding. Because of the iterative nature of QCA, it is possible and may be necessary to recalibrate the data following the analysis.

    This study has some limitations. Because the district-level effects were the same across the district, we could only assess the facility-level and community-level part of the ToC and could not assess the effect of the engagement, service strengthening and input into the medication procurement system, which was identified in the ToC. This will be expanded on in another paper which investigates the district-level implementation of the Nepal MHCP. In addition, although the PRIME MHCP was implemented in routine health services, a significant amount of support was provided by TPO Nepal, the NGO implementing the study. For example, they provided medication when the government had not yet started supplying this and they provided staff to assist with supervision, and directly implemented the community counselling component. As a result, most of the components of the MHCP were relatively well implemented in most of the facilities. However, as most inputs were available at similar times, it is difficult to determine which ones actually had an effect. Although the MHCP was piloted in two districts and there was regular supervision to ensure that interventions were delivered as intended, there were no formal fidelity assessments conducted prior to this study. This analysis also does not take into account some of the weaknesses of using longitudinal data, such as autocorrelation, non-stationarity and seasonality.50 It may be worth exploring how to combine statistical approaches for longitudinal data analysis with QCA, for example, by using time differencing QCA to look at the change in conditions over the time period of interest and using segmented linear regression to determine the presence or absence of the outcome. Another limitation is the use of pooled QCA which treats each time point as independent and does not take into account the between-clinic variation, and it may be possible to conduct a fixed-effects QCA as software are expanded.36 Lastly, QCA and the ToC did not take into account some of the features of complex adaptive systems such as tipping points, feedback loops, emergence or unintended consequences.51 Despite this, the approach does explore the multiple facility-level and community-level pathways required to achieve the outcome and therefore goes some of the way towards unpacking the black box of complex health interventions.

    Conclusions

    This study has shown that multiple packages of care are required to integrate mental health into primary care in low resource settings. Specifically, that a combination of formal manualised approaches to health worker detection and treatment, both psychological and pharmacological, referral pathways and trained community workers were necessary for high mental health service utilisation in West Chitwan. In addition, a combination of training, medication and either community awareness or use of community case finding was important. ToC can add value to QCA by identifying the causal pathways and the potential necessary and sufficient conditions which can be explored through QCA. This approach can help to shed light on the black box of complex multilevel interventions. Policy-makers who are planning integrated mental healthcare should ensure that mental healthcare plans are carefully constructed with multiple packages of care. These should include supply-side factors such as formalised approaches to health worker detection and treatment, training of health and community workers, supervision and medication supply. In addition, demand-side factors such as increasing awareness in the community and community case finding should be included.

    Acknowledgments

    We want to thank Anup Adhikari and the TPO Nepal research team for their involvement in data collection of the quarterly profiles, and Florian Breuer for assistance with the mathematical aspects of QCA.

    References

    1. 1.↵
      2. Saxena S ,
      3. Funk M ,
      4. Chisholm D
      . World health assembly adopts comprehensive mental health action plan 2013–2020. The Lancet 2013;381:1970–1.doi:10.1016/S0140-6736(13)61139-3
      OpenUrl
    2. 2.↵
      2. Saraceno B ,
      3. van Ommeren M ,
      4. Batniji R , et al
      . Barriers to improvement of mental health services in low-income and middle-income countries. The Lancet 2007;370:1164–74.doi:10.1016/S0140-6736(07)61263-X
      OpenUrl
    3. 3.↵
      2. Thornicroft G ,
      3. Tansella M
      . The balanced care model: the case for both hospital- and community-based mental healthcare. Br J Psychiatry 2013;202:246–8.doi:10.1192/bjp.bp.112.111377
      OpenUrlAbstract/FREE Full Text
    4. 4.↵
      2. Thornicroft G ,
      3. Tansella M
      . The balanced care model for global mental health. Psychol Med 2013;43:849–63.doi:10.1017/S0033291712001420
      OpenUrlCrossRefPubMed
    5. 5.↵
      2. Patel V ,
      3. Thornicroft G
      . Packages of care for mental, neurological, and substance use disorders in low- and middle-income countries: PLoS Medicine Series. PLoS Med 2009;6:e1000160.doi:10.1371/journal.pmed.1000160
      OpenUrlCrossRefPubMed
    6. 6.↵
      2. Patel V ,
      3. Simon G ,
      4. Chowdhary N , et al
      . Packages of care for depression in low- and middle-income countries. PLoS Med 2009;6:e1000159.doi:10.1371/journal.pmed.1000159
      OpenUrlCrossRefPubMed
    7. 7.↵
      2. Benegal V ,
      3. Chand PK ,
      4. Obot IS
      . Packages of care for alcohol use disorders in low- and middle-income countries. PLoS Med 2009;6:e1000170.doi:10.1371/journal.pmed.1000170
      OpenUrlPubMed
    8. 8.↵
      2. de Jesus MJ ,
      3. Razzouk D ,
      4. Thara R , et al
      . Packages of care for schizophrenia in low- and middle-income countries. PLoS Med 2009;6:e1000165.doi:10.1371/journal.pmed.1000165
    9. 9.↵
      2. Dua T ,
      3. Barbui C ,
      4. Clark N , et al
      . Evidence-based guidelines for mental, neurological, and substance use disorders in low- and middle-income countries: summary of WHO recommendations. PLoS Med 2011;8:e1001122.doi:10.1371/journal.pmed.1001122
      OpenUrlCrossRefPubMed
    10. 10.↵
      2. Shidhaye R
      . Implementation Science for closing the treatment gap for mental disorders by translating evidence base into practice: experiences from the PRIME project. Australas Psychiatry 2015;23(6 Suppl):35–7.doi:10.1177/1039856215609771
      OpenUrlCrossRefPubMed
    11. 11.↵
      2. Craig P ,
      3. Dieppe P ,
      4. Macintyre S , et al
      . Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655.doi:10.1136/bmj.a1655
      OpenUrlFREE Full Text
    12. 12.↵
      2. Lund C ,
      3. Tomlinson M ,
      4. De Silva M , et al
      . PRIME: a programme to reduce the treatment gap for mental disorders in five low- and middle-income countries. PLoS Med 2012;9:e1001359.doi:10.1371/journal.pmed.1001359
      OpenUrlCrossRefPubMed
    13. 13.↵
      2. Jordans MJ ,
      3. Luitel NP ,
      4. Pokhrel P , et al
      . Development and pilot testing of a mental healthcare plan in Nepal. Br J Psychiatry 2016;208 s21–28.doi:10.1192/bjp.bp.114.153718
      OpenUrlAbstract/FREE Full Text
    14. 14.↵
      2. Jordans MJ ,
      3. Luitel NP ,
      4. Tomlinson M , et al
      . Setting priorities for mental health care in Nepal: a formative study. BMC Psychiatry 2013;13:332.doi:10.1186/1471-244X-13-332
      OpenUrlCrossRefPubMed
    15. 15.↵
      2. Breuer E ,
      3. De Silva MJ ,
      4. Shidaye R , et al
      . Planning and evaluating mental health services in low- and middle-income countries using theory of change. Br J Psychiatry 2016;208:s55–62.doi:10.1192/bjp.bp.114.153841
      OpenUrlAbstract/FREE Full Text
    16. 16.↵
      2. De Silva MJ ,
      3. Breuer E ,
      4. Lee L , et al
      . Theory of Change: a theory-driven approach to enhance the medical research council's framework for complex interventions. Trials 2014;15:267.doi:10.1186/1745-6215-15-267
      OpenUrlCrossRefPubMed
    17. 17.↵
      2. Breuer E ,
      3. De Silva MJ ,
      4. Fekadu A , et al
      . Using workshops to develop theories of change in five low and middle income countries: lessons from the programme for improving mental health care (PRIME). Int J Ment Health Syst 2014;8:15.doi:10.1186/1752-4458-8-15
      OpenUrlCrossRefPubMed
    18. 18.↵
      2. Breuer E ,
      3. Lee L ,
      4. De Silva M , et al
      . Using theory of change to design and evaluate public health interventions: a systematic review. Implementation Science 2015;11:63.doi:10.1186/s13012-016-0422-6
      OpenUrl
    19. 19.↵
      2. Kane H ,
      3. Lewis MA ,
      4. Williams PA , et al
      . Using qualitative comparative analysis to understand and quantify translation and implementation. Transl Behav Med 2014;4:201–8.doi:10.1007/s13142-014-0251-6
      OpenUrl
    20. 20.↵
      2. De Silva MJ ,
      3. Rathod SD ,
      4. Hanlon C , et al
      . Evaluation of district mental healthcare plans: the PRIME consortium methodology. Br J Psychiatry 2016;208 s63–70.doi:10.1192/bjp.bp.114.153858
      OpenUrlAbstract/FREE Full Text
    21. 21.↵
      2. Luitel NP ,
      3. Jordans MJ ,
      4. Adhikari A , et al
      . Mental health care in Nepal: current situation and challenges for development of a district mental health care plan. Confl Health 2015;9:3.doi:10.1186/s13031-014-0030-5
      OpenUrl
    22. 22.↵
      2. Upadhaya N ,
      3. Jordans MJD ,
      4. Pokhrel R , et al
      . Current situations and future directions for mental health system governance in Nepal: findings from a qualitative study. Int J Ment Health Syst 2017;11:37.doi:10.1186/s13033-017-0145-3
      OpenUrl
    23. 23.↵
      1. Central Bureau of Statistics
      . National Poplulation and Housing Census 2011. In: Central Bureau of Statistics. Kathmandu Nepal: National Planning Comission Secretariat Government of Nepal, 2014.
    24. 24.↵
      1. World Health Organization
      . Mental Health Gap Action Programme Intervention Guide (mhGAP-IG) for mental, neurological and substance use disorders in non-specialized health settings. Geneva: WHO, 2010.
    25. 25.↵
      2. Chowdhary N ,
      3. Anand A ,
      4. Dimidjian S , et al
      . The Healthy Activity Program lay counsellor delivered treatment for severe depression in India: systematic development and randomised evaluation. Br J Psychiatry 2016;208:381-8.doi:10.1192/bjp.bp.114.161075
      OpenUrl
    26. 26.↵
      2. Dabholkar H ,
      3. Nadkarni A ,
      4. Velleman R
      . Counselling for Alcohol Problems CAP. Sangath, 2014.
    27. 27.↵
      2. Jordans MJ ,
      3. Kohrt BA ,
      4. Luitel NP , et al
      . Accuracy of proactive case finding for mental disorders by community informants in Nepal. Br J Psychiatry 2015;207:501–6.doi:10.1192/bjp.bp.113.141077
      OpenUrlAbstract/FREE Full Text
    28. 28.↵
      2. Jordans MJ ,
      3. Kohrt BA ,
      4. Luitel NP , et al
      . Proactive community case-finding to facilitate treatment seeking for mental disorders, Nepal. Bull World Health Organ 2017;95:531–6.doi:10.2471/BLT.16.189282
      OpenUrl
    29. 29.↵
      2. Ragin CC
      . Using qualitative comparative analysis to study causal complexity. Health Serv Res 1999;34 1225–39.
      OpenUrlCrossRefPubMedWeb of Science
    30. 30.↵
      2. Garson D
      . Case study analysis and QCA. Asheboro North Carolina: Statistical Associates Publishing, 2016.
    31. 31.↵
      2. Warren J ,
      3. Wistow J ,
      4. Bambra C
      . Applying qualitative comparative analysis (QCA) in public health: a case study of a health improvement service for long-term incapacity benefit recipients. J Public Health 2014;36:126–33.doi:10.1093/pubmed/fdt047
      OpenUrlCrossRefPubMedWeb of Science
    32. 32.↵
      2. McAlearney AS ,
      3. Walker D ,
      4. Moss AD , et al
      . Using qualitative comparative analysis of key informant interviews in health services research: enhancing a study of adjuvant therapy use in breast cancer care. Med Care 2016;54:400–5.doi:10.1097/MLR.0000000000000503
      OpenUrl
    33. 33.↵
      2. Longest KC ,
      3. Vaisey S
      . Fuzzy: a program for performing qualitative comparative analyses (QCA) in Stata. The Stata Journal: Promoting communications on statistics and Stata 2008;8:79–104.doi:10.1177/1536867X0800800106
      OpenUrl
    34. 34.↵
      1. STATA program
      . 13 version. College Station: Statcorp, 2013.
    35. 35.↵
      2. Ragin C
      . Fuzzy set analysis: calibration vs. measurement. In: Box-Steffensmeier JM , Brady HE , Collier D , eds. The Oxford handbook of political methodology. Oxford New York: Oxford University Press, 2008: 880.
    36. 36.↵
      2. Hino A
      . Time-Series QCA. 理論と方法 2009;24:247–65.
      OpenUrl
    37. 37.↵
      1. COMPASSS: Software
      . fs/QCA [program]: 2.5 version. Irvine,CA: University of California, 2014.
    38. 38.↵
      2. Rihoux B ,
      3. Ragin CC
      . Configurational comparative methods: Qualitative Comparative Analysis QCA. and related techniques: Sage Publications, 2008.
    39. 39.↵
      2. Jenkins R ,
      3. Othieno C ,
      4. Okeyo S , et al
      . Short structured general mental health in service training programme in Kenya improves patient health and social outcomes but not detection of mental health problems—a pragmatic cluster randomised controlled trial. Int J Ment Health Syst 2013;7:25.doi:10.1186/1752-4458-7-25
      OpenUrlCrossRefPubMed
    40. 40.↵
      2. Kauye F ,
      3. Jenkins R ,
      4. Rahman A
      . Training primary health care workers in mental health and its impact on diagnoses of common mental disorders in primary care of a developing country, Malawi: a cluster-randomized controlled trial. Psychol Med 2014;44:657–66.doi:10.1017/S0033291713001141
      OpenUrl
    41. 41.↵
      2. Goncalves DA ,
      3. Fortes S ,
      4. Campos M , et al
      . Evaluation of a mental health training intervention for multidisciplinary teams in primary care in Brazil: a pre- and posttest study. Gen Hosp Psychiatry 2013;35:304–8.doi:10.1016/j.genhosppsych.2013.01.003
      OpenUrlCrossRefPubMed
    42. 42.↵
      2. Painuly N ,
      3. Sharan P
      . Effectiveness of training of non-mental health care providers in mental health in low- and middle-income countries: a systematic review. Primary Care and Community Psychiatry 2008;13:83–9.
      OpenUrl
    43. 43.↵
      2. Brenman NF ,
      3. Luitel NP ,
      4. Mall S , et al
      . Demand and access to mental health services: a qualitative formative study in Nepal. BMC Int Health Hum Rights 2014;14:22.doi:10.1186/1472-698X-14-22
      OpenUrl
    44. 44.↵
      2. Luitel NP ,
      3. Jordans MJD ,
      4. Kohrt BA , et al
      . Treatment gap and barriers for mental health care: a cross-sectional community survey in Nepal. PLoS One 2017;12:e0183223.doi:10.1371/journal.pone.0183223
      OpenUrl
    45. 45.↵
      2. Althaus F ,
      3. Paroz S ,
      4. Hugli O , et al
      . Effectiveness of interventions targeting frequent users of emergency departments: a systematic review. Ann Emerg Med 2011;58:41–52.doi:10.1016/j.annemergmed.2011.03.007
      OpenUrlCrossRefPubMed
    46. 46.↵
      2. Kakuma R ,
      3. Minas H ,
      4. van Ginneken N , et al
      . Human resources for mental health care: current situation and strategies for action. The Lancet 2011;378:1654–63.doi:10.1016/S0140-6736(11)61093-3
      OpenUrl
    47. 47.↵
      2. Dill JS ,
      3. Chuang E ,
      4. Morgan JC
      . Healthcare organization-education partnerships and career ladder programs for health care workers. Soc Sci Med 2014;122:63–71.doi:10.1016/j.socscimed.2014.10.021
      OpenUrl
    48. 48.↵
      2. Goicolea I ,
      3. Vives-Cases C ,
      4. Hurtig AK , et al
      . Mechanisms that trigger a good health-care response to intimate partner violence in Spain. Combining realist evaluation and qualitative comparative analysis approaches. PLoS One 2015;10:e0135167.doi:10.1371/journal.pone.0135167
      OpenUrlCrossRefPubMed
    49. 49.↵
      2. Yamasaki S ,
      3. Rihoux B
      . 6 A Commented Review of Applications 2009:123–46.
    50. 50.↵
      2. Lagarde M ,
      3. do Hto
      . How to do (or not to do)… Assessing the impact of a policy change with routine longitudinal data. Health Policy Plan 2012;27:76–83.doi:10.1093/heapol/czr004
      OpenUrlCrossRefPubMedWeb of Science
    51. 51.↵
      2. Glouberman S ,
      3. Ph D ,
      4. Zimmerman B
      . Complicated and complex systems: what would successful reform of Medicare look like? 2002.

    Footnotes

    • Handling editor Valery Ridde

    • Contributors EB was responsible for the conception and design of the study with substantial contributions from MDS, CL and BM. EB, CL, MDS, NL and MJ contributed to the design of the data collection instruments. The data were collected as part of the PRIME consortium evaluation in Nepal led by MJ and NL and assisted by PS. PS, NL and MJ assisted with the interpretation of the data and gave information about the context. EB conducted the analysis with support from CL and wrote the first draft of the paper. All authors were involved in the interpretation of the results, revising the manuscript critically for important intellectual content and reviewing the manuscript prior to submission.

    • Funding This study is an output of the PRogramme for Improving Mental health carE (PRIME). This work was supported by the UK Department for International Development (201446).

    • Disclaimer The views expressed do not necessarily reflect the UK Government’s official policies.

    • Competing interests None declared.

    • Patient consent Not required.

    • Ethics approval Permission was obtained from the managers of all 10 facilities to obtain facility-level data. This study was approved by Faculty of Health Sciences Human Research Ethics Committee, University of Cape Town (HREC 412/2011 and 247/2013) and the Nepal Health Research Council (NHRC 10/2013) and conformed to the principles embodied in the Declaration of Helsinki.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The raw data used during the current study are available from the corresponding author on reasonable request. The calibrated data analysed during this study is included in the online supplementary files. Additional data from PRIME can be accessed on request from www.prime.uct.ac.za in line with PRIME’s data sharing policies.