Applying complexity theory: A review to inform evaluation design
Section snippets
Methods
This study provides a narrative thematic review of identified academic journal literature (Dixon-Woods et al., 2005, Mays et al., 2005) related to complexity theory and evaluation. This review draws upon 46 articles in peer-reviewed journals identified from a search of bibliographic databases (including Scopus, Web of Knowledge, Social Service Abstracts, Sociological abstracts), limited to English language. Search terms were: complexity theory or complex adaptive system or CAS or soft system or
Results
The following themes were identified from the reviewed literature and are discussed in detail below:
- •
Developing an understanding of system.
- •
Attractors, emergence and other complexity concerns.
- •
Defining appropriate level and unit of analysis.
- •
Timing of evaluations.
- •
Participatory methods.
- •
Case study and comparison designs.
- •
Multiple and mixed methods.
- •
Layering theory to guide evaluation.
Conclusion
This paper has sought to identify a range of implications of a complexity frame of reference for policy and programme evaluation. Though restricted to a small number of papers in the academic literature, some common themes have been identified. The themes can be broken into two groups. The first group identifies implications of applying complexity theory for defining the purpose, scope and relevant units of analysis for evaluation activity. The second group of themes consider method
Acknowledgements
This project is supported by the Marsden Fund Council from Government funding, administered by the Royal Society of New Zealand (MAU1107).
Mat Walton is a lecturer in the School of Health and Social Services, Massey University, New Zealand. His research focuses on public health policy design and evaluation, the application of complexity theory and policy interaction.
References (78)
- et al.
A qualitative comparative analysis of factors associated with trends in narrowing health inequalities in England
Social Science and Medicine
(2011) - et al.
Economic evaluation to inform health care decision-making: Promise, pitfalls and a proposal for an alternative path
Social Science and Medicine
(2011) - et al.
Systems thinking
Evaluation and Program Planning
(2008) Complexity and reflexivity: Two important issues for economic evaluation in health care
Social Science and Medicine
(2007)Social multi-criteria evaluation: Methodological foundations and operational consequences
European Journal of Operational Research
(2004)Development as a complex process of change: Conception and analysis of projects, programs and policies
International Journal of Educational Development
(2010)- et al.
Assessing policies from a systems perspecitve – Experiences with applied innovation systems analysis and implications for policy evaluation
Forest Policy and Economics
(2006) - et al.
Why performance-based contracting failed in Uganda – An “open-box” evaluation of a complex health system intervention
Social Science and Medicine
(2012) - et al.
Using complexity theory to analyse the organisational response to resurgent tuberculosis across London
Social Science and Medicine
(2013) - et al.
Evaluating health systems strengthening interventions in low-income and middle-income countries: Are we asking the right questions?
Health Policy and Planning
(2012)
Case study research: The view from complexity science
Qualitative Health Research
Evidence understanding and complexity: Evaluation in non-linear systems
Evaluation
Wicked comparisons: Reflections on cross-national research about health inequalities in the UK
Evaluation
Using qualitative comparative analysis to understand complex policy problems
Evaluation
Selecting a change and evaluating its impact on the performance of a complex adaptive health care delivery system
Clinical Interventions in Aging
Developing an evaluation framework for crosscutting policy goals: The accessibility policy assessment tool
Environment and Planning A
Approaches to researching complexity in public management
Evaluating action learning: A critical realist complex network theory approach
Action Learning: Research and Practice
Evaluation in complex governance arenas: The potential of large system action research
Complexity configurations and cases
Theory Culture Society
Applying social science: The role of social research in politics, policy and practice
Evaluating complex social interventions in a complex world
Evaluation
Complexity theory and the social sciences: The state of the art
Evaluation and negotiated order: Developing the application of complexity theory
Evaluation
Complexity and life
Theory Culture Society
Complexity and whole-system change programmes
Journal of Health Services Research and Policy
The evaluation of health and social care partnerships: An analysis of approaches and synthesis for the future
Health and Social Care in the Community
Synthesising qualitative and quantitative evidence: A review of possible methods
Journal of Health Services Research and Policy
Dealing with complexity: Theory of change evaluation and the full service extended schools initiative
International Journal of Research and Method in Education
Applying complexity theory to New Zealand public policy: Principles for practice
Policy Quarterly
Food fight: The battle over redefining competitive foods
The Journal of School Health
Systems theory and complexity introduction
Democracy and Nature
Emergence as a construct: History and issues
Emergence
Knowledge theories can inform evaluation practice: What can a complexity lens add?
New Directions for Evaluation
Methods for exploring implementation variation and local context within a cluster randomised community intervention trial
Journal of Epidemiology and Community Health
Complexity theory and evaluation in public management
Public Management Review
Researching a local heroin market as a complex adaptive system
American Journal of Community Psychology
Nonlinearity in social service evaluation: A primer on agent-based modeling
Social Work Research
Cited by (82)
How guest-host interactions affect consumer experiences in the sharing economy: New evidence from a configurational analysis based on consumer reviews
2022, Decision Support SystemsCitation Excerpt :The nonlinear interaction of components implies that change in one component may insignificantly or hugely impact the entire system. Hence, understanding the system requires examining it as a whole, not its individual components [34]. The essence of complexity theory is its more in-depth explanation of how consumers evaluate services and outcomes [35].
Evaluating the integration of strategic priorities within a complex research-for-development funding program
2021, Evaluation and Program PlanningThe development and psychometric properties of an educational development impact questionnaire
2021, Studies in Educational EvaluationCitation Excerpt :Together these findings are important for understanding the types of services accessed by different instructor groups. To understand complex systems as a whole it is necessary to focus the evaluation approach across multiple levels over time (Walton, 2014). According to our outcomes framework, the common five factors map to four out of five long-term impacts (Fig. 5) which help us to consider the usefulness of the EDIQ for suggesting impact at the institutional level.
Evaluation of public health interventions from a complex systems perspective: A research methods review
2021, Social Science and MedicineCitation Excerpt :We consulted experts (n = 32) with an interest and expertise in systems-oriented public health research. We also conducted citation searches of relevant published systematic reviews beginning with two we had pre-identified (Carey et al., 2015; Walton, 2014). For our electronic search, we adapted search terms (Supporting Information 2) from those reviews and searched Scopus, MEDLINE and Web of Science, searching after the time period covered by the published systematic reviews (Carey et al., 2015; Walton, 2014).
The Policy and Regulatory Engagement with Corruption: Insights from Complexity Theory
2022, European Journal of Risk RegulationPost-project sustainability evaluation by using a complex systems approach: A framework and an application to a revolving loan fund (microfinance) project in Indonesia
2024, Systems Research and Behavioral Science
Mat Walton is a lecturer in the School of Health and Social Services, Massey University, New Zealand. His research focuses on public health policy design and evaluation, the application of complexity theory and policy interaction.