Article Text
PDF
Abstract
Pharmaceutical industry-led access programmes are increasing in number and scope worldwide. We present a new standardised framework for evaluation of these programmes that includes three components: a taxonomy of 11 access programme strategies; a series of logic models, one for each strategy and a set of measurement indicators. The logic models describe pathways of potential programme impact. Concepts relevant across a broad range of strategies were prioritised for inclusion in logic models to ensure consistency and to facilitate synthesis and learning across programmes. Each concept has at least one corresponding measurement indicator with metadata that includes the definition, details on how it should be measured and recommended data sources. The framework establishes a shared language for the collection and reporting of meaningful industry-led access programme information. Broad adoption by programme developers and implementing partners in the for-profit sector and beyond could facilitate shared learning on effective strategies and best practices.
- health systems evaluation
- public health
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Summary box
Pharmaceutical industry-led access programmes are increasing in number and scope worldwide, but a lack of evidence on program performance is hindering shared learning and accountability.
Lack of standard publicly available performance assessment tools that focus specifically on industry-led access programmes presents a key barrier to assessing programme achievements and ensuring company accountability for commitments made.
This paper presents a new framework for evaluation of industry-led access programmes.
The framework establishes standards for evaluation that can be applied across programmes and serves as a shared language for the collection and reporting of meaningful programme information.
Companies and non-profit organisations can use the framework to guide the design of their access programmes and evaluation systems for those programmes.
Broad adoption of the framework by programme developers and implementing partners including the for-profit sector could facilitate accountability and promote increased investments in effective strategies.
Introduction
Pharmaceutical industry-led access programmes are increasing in number and scope worldwide, driven by expanding commitments by private pharmaceutical companies.1 These programmes aim to address access barriers using a variety of health system approaches,2 including medicine donation,3 4 supply chain strengthening,5–7 healthcare provider training8 and community outreach.9 Detailed information on industry-led access programmes activities and impacts is often not publicly available,1 making it difficult to assess whether expanding programmes will translate into stronger health systems, increased patient access and improved population health. Lack of standardised reporting also hampers the public’s understanding of programmes and limits potential opportunities to learn about effective approaches and best practices.
More than 20 large biopharmaceutical companies, in partnership with the World Bank and the Union for International Cancer Control (UICC), recently committed to expanding their efforts to improve access for non-communicable diseases (NCD) in low-income and middle-income countries (LMICs) as part of the Access Accelerated initiative.10 11 Access Accelerated programmes go beyond a narrow focus on medicines, and aim to address the full range of health system barriers to access. As part of the initiative, members agreed to measure and publicly report on their programmes, with a focus on their individual and joint contributions to achieving the sustainable development goals (SDGs), in particular Target 3.4 related to addressing NCD.12 13
Private sector measurement of social impact has received increased interest in recent years and more than 100 relevant measurement tools exist.14–16 However, most of these tools lack a transparent methodology and many are proprietary and require payment for use.17 There are no standard publicly available performance assessment tools that focus specifically on access programmes. This presents a key barrier to assessing programme achievements and ensuring company accountability to commitments.18 There is an opportunity and a need to develop a new framework for the evaluation of industry-led access programmes.
In this paper, we present a standardised framework for the evaluation of access programmes that was developed as part of the Access Accelerated initiative, but is not limited to programmes focused on NCDs or programmes led by private pharmaceutical companies. We first describe the principles that guided the development of the framework. We then describe the three main components of the framework: a taxonomy of 11 access programme strategies; a series of logic models, one for each strategy and a set of clearly defined measurement indicators, which include measures of programme performance and achievements. We conclude with a discussion of the framework’s potential uses and future evolution.
Principles of framework development
Developing an evaluation framework requires making value-based judgements, whether explicit or implicit. For example, with respect to content, whether to prioritise medicine access or population health as the primary measure of impact requires deliberation on what should constitute the ultimate goals of programme activities. Programme designers and external stakeholders might reasonably disagree on goals; developing a standardised approach does not require choosing a side in the debate—a flexible approach is often best—but it does require careful consideration of the relevant values.
Four core principles guided the development of the framework, informing decisions on process and content. The process was guided by the principles of independence from industry, methodological rigour and transparency to the public and content decisions were guided by the principle that public health goals are of primary importance. These guiding principles are detailed below.
Independence from industry
The framework was developed with independence from the pharmaceutical industry, which is fundamental to its legitimacy. To ensure a practical approach, we consulted regularly with representatives from Access Accelerated companies, as well as the World Bank and UICC. We made all final decisions on the design of the framework. All legal contracts governing the development of the framework are publicly available.19
Methodological rigour
The framework was designed to facilitate rigorous evaluation by users. The taxonomy of programme strategies was structured to be thorough and precise. The logic models were constructed according to a standard ‘theory of change’ approach to clarify primary pathways through which strategies might achieve impact.20 21 The indicators were selected to balance high construct and content validity with the potential for high-quality data collection.
Transparency to the public
The framework was designed with a primary purpose of facilitating greater transparency in reporting on pharmaceutical industry-led access programmes. Transparency has several benefits, including generating opportunities for learning and potentially strengthening accountability. Standardising indicators facilitates transparency by establishing clear and agreed on definitions and meanings, that is, by ensuring that shared information is intelligible to the public and other audiences.18 22 While designing the framework, we elicited feedback from potential users and public stakeholders to refine the terminology used and strengthen common understanding.
Public health goals
Public health goals were prioritised during the development of framework content, consistent with the pharmaceutical industry’s commitment to the SDGs.12 The WHO has identified the ultimate goals of a health system as population health; financial risk protection and responsiveness (including patient satisfaction).23 As described below, the framework’s impact indicators align with the WHO goals; intermediate access indicators are also included as pathways to these goals.
Framework components
The framework includes three main components: (1) a taxonomy of 11 strategies that describes common approaches used by pharmaceutical industry-led access programmes; (2) a series of logic models—one per strategy—detailing the primary pathways through which programmes might achieve impact and (3) a set of clearly defined indicators for measuring programme activities and achievements.
Taxonomy of programme strategies
The core foundation of the framework is a taxonomy that can be used to categorise access programmes into 11 broad strategies (table 1). The structure of the taxonomy was derived in part from existing access and health system frameworks, including Bigdeli et al ,2 WHO’s Equitable Access to Essential Medicines framework,24 and the health reform framework developed by Roberts et al.25 A preliminary list of strategies was tested and refined during a process that included a review of existing industry-led access programmes.26 By design, the taxonomy simplifies the organisation of a tremendous variety of programme approaches. This simplification allows for the application of standardised indicators specific to each strategy.
Taxonomy of programme strategies
The 11 strategies are organised in four broad categories: (1) community strategies; (2) health system strategies; (3) medicine production strategies and (4) medicine price strategies. Community strategies focus on communities and patients and aim to increase awareness of symptoms of disease or treatment options, or to link patients to available care. Health system strategies focus on strengthening existing health services, delivering parallel services directly to patients, strengthening medicine and product supply chains, expanding access to healthcare financing, or harmonising regulation and legislation processes. Medicine production strategies focus on increasing production of medicines by increasing manufacturing capacity, strengthening product development research, or easing legal barriers to production through licensing agreements. Finally, medicine pricing strategies aim to improve affordability through donations or price schemes. Most access programmes pursue more than one strategy, and the framework is designed to allow for this.
Logic models
Each of the 11 strategies has a corresponding logic model that describes the primary pathways of potential programme impact. The logic models were structured in accordance with standard formats, such as those recommended by the Centers for Disease Control and Prevention20 and the US Agency for International Development.21 Flowing from left to right, each model includes five columns: inputs, activities, outputs, outcomes (short-term and long-term) and impacts (see figure 1 for an example; the full set of logic models is provided in the online supplementary file). Connexions across columns indicate hypothetical if-then statements, for example, if programme outputs are distributed to intended beneficiaries then short-term outcomes will be achieved. Boxes in each column indicate concepts along the causal pathways from programme implementation to impact; each concept is operationalised with a corresponding measurement indicator as described in the next section.
Supplemental material
Example logic model.
The concepts included in the logic models were not intended to be exhaustive; the diversity of programme approaches makes this impossible. The intention was rather to identify priority concepts that are likely to be relevant across a broad range of approaches, to ensure a basic level of consistency and to facilitate synthesis and learning across programs. For individual programme, developers are encouraged to use the framework logic models as a starting point, and to add additional concepts as appropriate. Wherever appropriate, concepts represented in the logic models were repeated across the 11-strategy series. Most repeated concepts have been previously identified as important in the literature on access to medicines including availability, appropriate use and accessibility.27 For example, availability of medicines at outlets has been identified as a key dimension of access22 and is included as an outcome in six of the 11 logic models. Input concepts are identical across all logic models, as are impact concepts. Inputs include the value of resources put into the programme and staff time devoted to the programme. Impacts mirror the WHO goals of a health system: population health, financial risk protection and patient satisfaction (a key aspect of responsiveness).23
Indicators
Each concept included in the logic models has at least one corresponding measurement indicator. For example, ‘population exposed’ is a concept included in the community awareness and linkage to care logic model that has a corresponding indicator, ‘population exposed to community communication activities’ with the definition, ‘number of population reached through a community awareness campaign’ (table 2). Each indicator has a table of metadata that includes the definition, details on how it should be measured and recommended data sources. When appropriate, metadata tables also include suggestions for stratification by demographic characteristics, equity dimensions (gender, geographical location and income), health sector (private, public and faith-based) or level of care (primary, secondary and tertiary). The full set of 72 indicators is organised in a data dictionary (included in the online supplementary file).
Example indicator metadata
Many indicators correspond to multiple logic models, allowing for potential comparison and synthesis across programmes that employ different strategies. For example, ‘availability of medicines at outlets’ is a common indicator for supply chain strengthening, medicine donation and price scheme strategies. Indicators were selected based on several criteria, including high construct and criterion validity; feasibility of collecting high-quality data at low cost and existence of validated data collection instruments.28 For example, ‘percentage of outlets with medicine in stock’ was selected as an indicator for measuring medicine availability because its high construct and criterion validity have been shown, and because the feasibility of high quality data collection at relatively low cost has been demonstrated.29 For each indicator, we recommend that programme data be collected and reported in a manner that allows for disaggregation by relevant subgroups (eg, socioeconomic status, gender) to assess equity.
Using the framework
We developed a standardised framework for evaluation of pharmaceutical industry-led access programmes, guided by the principles of independence, rigour and transparency, and placing primary importance on public health goals. The framework establishes a shared language for the collection and reporting of meaningful programme information.
Our approach is distinct from existing social performance tools used by the pharmaceutical industry, particularly the Access to Medicine Index.30 The Index includes commercial activities in addition to social programmes in its metrics, and measures performance at the company-level rather than the programme level. Our approach is similar to a framework recently developed for evaluating alcohol industry-led social programme, which also includes a series of logic models and a set of indicators.31
We aim to facilitate shared learning and improved programme performance, and we anticipate several audiences for this work. First, the framework can be used by companies and non-profit organisations to guide the design of their access programmes and evaluation systems for those programmes. We encourage users of the framework to commit to transparency and publish their work in a public forum. We have developed a public platform for transparent reporting on access programmes, the Access Observatory, which can be used by any organisation working in this area.32 Second, the framework can be used as a conceptual model by researchers and policymakers who aim to understand and analyse these programmes.
Finally, local and global stakeholders can incorporate the framework and programme information into their processes for holding companies and implementing organisations accountable. WHO has identified key considerations that should be taken into account when assessing industry-led medicines and health technologies access programmes; among these is whether programmes have clearly defined performance targets.33 In addition, civil society and health systems researchers have called for greater clarity on programme adherence to regulations and guidelines (eg, national treatment guidelines) and on assignment of responsibilities among the various components of the programme partners for implementation activities.34 Our framework can guide countries, civil society organisations and other stakeholders in their dialogue with for-profit companies and non-profit organisations about alignment with national priorities, target setting and reporting. Establishing clear expectations among all stakeholders by defining commonly agreed on indicators and reporting mechanisms is critical for increasing transparency and accountability of for-profit companies and non-profit organisations.
In 2017, 63 pharmaceutical industry-led access programme operating in 103 LMICs applied the framework for the first time. Individual reports for each programme are available on the Access Observatory website,32 and the Access Observatory 2018 Report includes an analysis of the full set of programmes.35 Around one-third of programmes reported indicator data into the Access Observatory in the first year. For many programmes, building the systems needed for data collection and management will be a process that takes time. Looking forward, the framework can play an important role in facilitating rigorous evaluation and transparent reporting on industry-led access programme, but this will require continued engagement on the part of the pharmaceutical industry and global health stakeholders.
The framework has important limitations. First, we focused on capturing the full breadth of strategies that the pharmaceutical industry is using in their efforts to increase access. For practical purposes, to ensure that the framework is usable, capturing this breadth may have limited the depth of the logic models and data dictionary that could be possible. We encourage users of the framework to further refine our tools to best fit their purposes. Second, the framework design was informed by a review of existing industry-led programmes operating in LMICs. Our intention is that the framework is general enough to be applied to non-industry programmes, programmes in high-income countries and programmes focused on any type of health condition. However, more work may be needed to adapt the framework to this type of more general use. Finally, while we set out clear principles at the start of developing the framework, users of the framework may apply their own set of principles, which may be different from ours. We encourage all users to adopt a principle of transparency and clearly state their other principles.
We expect that the framework will evolve in several key areas in the future. First, the taxonomy of strategies will likely expand to include new and innovative programme approaches. Each new strategy will be accompanied by a new logic model and set of indicators. Second, the indicator set will likely change as it becomes clearer which data are most feasible to collect. This may entail dropping some indicators and adding others while maintaining high content validity. Third, while the framework was developed as part of Access Accelerated, the strategies, logic models and indicators are applicable to access programmes more generally, not only those led by industry or focused on NCDs. We anticipate that the components of the framework will evolve as the full diversity of non-industry programmes becomes clearer. Stakeholder commitment and resources are required to further develop the framework.
Our experience suggests that programme teams often have the capacity and resources to collect and report information on input and output indicators, but less often for outcome and impact indicators, which in most instances are more challenging and costlier to collect. As a result, we expect that for many programmes, at least early on, reported data will describe the scope of programme activities but information on programme outcomes and impacts will be limited. Since programme outcome and impact data are critical to determine whether the programme requires changes we expect that programme funders as well as beneficiaries will demand this information. As a result, rigorous impact evaluations would receive a higher priority in the future.
Many existing industry-led access programmes are pursuing similar strategies and pooling resources across programmes may make it easier to conduct resource intensive evaluations.18 In addition, the development of new methodologies that balance rigour with practicality and less costly data collection could produce increased opportunities for efficient evaluation of programme outcomes and impacts.36 The framework creates a shared language that should help in identifying potential synergies and opportunities across programmes.
Conclusions
Improving access to essential medicines is a key global health priority, identified in SDG Target 3.8. The pharmaceutical industry and other stakeholders are increasing their commitments to addressing this issue. Measurement standards will be important for understanding the impacts of these efforts, and for steering resources to strategies that are proven to work. The framework we present in this paper is the first to establish standards for evaluation that can be applied across pharmaceutical industry-led access programme. Other industries have also been recognised as key contributors to the SDGs,37 and our work may provide a roadmap for similar frameworks focused on other goals, for example environmental sustainability. In the coming years, efforts in this area will need to be expanded to increase programme efficiency, sustainability and accountability to contribute to achieving important global development goals.
Supplemental material
Footnotes
Handling editor Seye Abimbola
Contributors All authors were involved in creating the measurement framework. PCR wrote the first draft of the manuscript. All the authors were involved in substantially reviewing and revising the manuscript. All the authors have read and approved the final draft of the manuscript.
Funding This work was funded through a grant from Access Accelerated, an initiative of more than 20 global biopharmaceutical companies in partnership with the World Bank and Union of International Cancer Control that seeks to reduce barriers to prevention, treatment and care for non-communicable diseases in low-income and middle-income countries. The funder had no role in final decisions on the design of the framework nor in the writing of the manuscript or the decision to publish. The Master Services Agreement governing this work is available at: http://sites.bu.edu/evaluatingaccess-accessaccelerated/agreements/.
Competing interests All authors report grants from Access Accelerated, during the conduct of the submitted work. PCR reports grants from Sandoz International GmBH and Gilead Sciences, outside the submitted work. ROL reports non-financial support from Novartis International AG and grants from Sandoz International GmBH and Gilead Sciences, outside the submitted work. NS reports a grant from MSD for Mothers, outside the submitted work. PA report a grant from Sandoz International GmBH, outside the submitted work. EHL reports personal fees from Vertex Pharmaceuticals, outside the submitted work. VJW reports grants from Sandoz International GmBH and Gilead Sciences, outside the submitted work.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data are available at www.accessobservatory.org.