Applying complexity theory: A review to inform evaluation design

doi:10.1016/j.evalprogplan.2014.04.002

Evaluation and Program Planning

Volume 45, August 2014, Pages 119-126

https://doi.org/10.1016/j.evalprogplan.2014.04.002 Get rights and content

Highlights

•
Reviews use and discussion of complexity theory in evaluation.
•
Identifies eight themes related to application of complexity theory for evaluation.
•
Themes can be used to guide evaluation design discussion.

Abstract

Complexity theory has increasingly been discussed and applied within evaluation literature over the past decade. This article reviews the discussion and use of complexity theory within academic journal literature. The aim is to identify the issues to be considered when applying complexity theory to evaluation. Reviewing 46 articles, two groups of themes are identified. The first group considers implications of applying complexity theory concepts for defining evaluation purpose, scope and units of analysis. The second group of themes consider methodology and method. Results provide a starting point for a configuration of an evaluation approach consistent with complexity theory, whilst also identifying a number of design considerations to be resolved within evaluation planning.

Section snippets

Methods

This study provides a narrative thematic review of identified academic journal literature (Dixon-Woods et al., 2005, Mays et al., 2005) related to complexity theory and evaluation. This review draws upon 46 articles in peer-reviewed journals identified from a search of bibliographic databases (including Scopus, Web of Knowledge, Social Service Abstracts, Sociological abstracts), limited to English language. Search terms were: complexity theory or complex adaptive system or CAS or soft system or

Results

The following themes were identified from the reviewed literature and are discussed in detail below:

•
Developing an understanding of system.
•
Attractors, emergence and other complexity concerns.
•
Defining appropriate level and unit of analysis.
•
Timing of evaluations.
•
Participatory methods.
•
Case study and comparison designs.
•
Multiple and mixed methods.
•
Layering theory to guide evaluation.

Conclusion

This paper has sought to identify a range of implications of a complexity frame of reference for policy and programme evaluation. Though restricted to a small number of papers in the academic literature, some common themes have been identified. The themes can be broken into two groups. The first group identifies implications of applying complexity theory for defining the purpose, scope and relevant units of analysis for evaluation activity. The second group of themes consider method

Acknowledgements

This project is supported by the Marsden Fund Council from Government funding, administered by the Royal Society of New Zealand (MAU1107).

Mat Walton is a lecturer in the School of Health and Social Services, Massey University, New Zealand. His research focuses on public health policy design and evaluation, the application of complexity theory and policy interaction.

References (78)

T. Blackman et al.
A qualitative comparative analysis of factors associated with trends in narrowing health inequalities in England
Social Science and Medicine
(2011)
A. Brousselle et al.
Economic evaluation to inform health care decision-making: Promise, pitfalls and a proposal for an alternative path
Social Science and Medicine
(2011)
D. Cabrera et al.
Systems thinking
Evaluation and Program Planning
(2008)
C. Lessard
Complexity and reflexivity: Two important issues for economic evaluation in health care
Social Science and Medicine
(2007)
G. Munda
Social multi-criteria evaluation: Methodological foundations and operational consequences
European Journal of Operational Research
(2004)
B.H. Nordtveit
Development as a complex process of change: Conception and analysis of projects, programs and policies
International Journal of Educational Development
(2010)
E. Rametsteiner et al.
Assessing policies from a systems perspecitve – Experiences with applied innovation systems analysis and implications for policy evaluation
Forest Policy and Economics
(2006)
F. Ssengooba et al.
Why performance-based contracting failed in Uganda – An “open-box” evaluation of a complex health system intervention
Social Science and Medicine
(2012)
S. Trenholm et al.
Using complexity theory to analyse the organisational response to resurgent tuberculosis across London
Social Science and Medicine
(2013)
T. Adam et al.
Evaluating health systems strengthening interventions in low-income and middle-income countries: Are we asking the right questions?
Health Policy and Planning
(2012)

R.A. Anderson et al.

Case study research: The view from complexity science

Qualitative Health Research

(2005)

M. Barnes et al.

Evidence understanding and complexity: Evaluation in non-linear systems

Evaluation

(2003)

T. Blackman et al.

Wicked comparisons: Reflections on cross-national research about health inequalities in the UK

Evaluation

(2010)

T. Blackman et al.

Using qualitative comparative analysis to understand complex policy problems

Evaluation

(2013)

M.A. Boustani et al.

Selecting a change and evaluating its impact on the performance of a complex adaptive health care delivery system

Clinical Interventions in Aging

(2010)

G. Bristow et al.

Developing an evaluation framework for crosscutting policy goals: The accessibility policy assessment tool

Environment and Planning A

(2009)

J.-M. Buijs et al.

Approaches to researching complexity in public management

J.G. Burgoyne

Evaluating action learning: A critical realist complex network theory approach

Action Learning: Research and Practice

(2010)

D. Burns

Evaluation in complex governance arenas: The potential of large system action research

D. Byrne

Complexity configurations and cases

Theory Culture Society

(2005)

D. Byrne

Applying social science: The role of social research in politics, policy and practice

(2011)

D. Byrne

Evaluating complex social interventions in a complex world

Evaluation

(2013)

D. Byrne et al.

Complexity theory and the social sciences: The state of the art

(2014)

G. Callaghan

Evaluation and negotiated order: Developing the application of complexity theory

Evaluation

(2008)

F. Capra

Complexity and life

Theory Culture Society

(2005)

B. Dattée et al.

Complexity and whole-system change programmes

Journal of Health Services Research and Policy

(2010)

H. Dickinson

The evaluation of health and social care partnerships: An analysis of approaches and synthesis for the future

Health and Social Care in the Community

(2006)

M. Dixon-Woods et al.

Synthesising qualitative and quantitative evidence: A review of possible methods

Journal of Health Services Research and Policy

(2005)

A. Dyson et al.

Dealing with complexity: Theory of change evaluation and the full service extended schools initiative

International Journal of Research and Method in Education

(2010)

E. Eppel et al.

Applying complexity theory to New Zealand public policy: Principles for practice

Policy Quarterly

(2011)

S. Fleischhacker

Food fight: The battle over redefining competitive foods

The Journal of School Health

(2007)

A. Gare

Systems theory and complexity introduction

Democracy and Nature

(2000)

J. Goldstein

Emergence as a construct: History and issues

Emergence

(1999)

P. Hawe et al.

Knowledge theories can inform evaluation practice: What can a complexity lens add?

New Directions for Evaluation

(2009)

P. Hawe et al.

Methods for exploring implementation variation and local context within a cluster randomised community intervention trial

Journal of Epidemiology and Community Health

(2004)

P. Haynes

Complexity theory and evaluation in public management

Public Management Review

(2008)

L.D. Hoffer et al.

Researching a local heroin market as a complex adaptive system

American Journal of Community Psychology

(2009)

N. Israel et al.

Nonlinearity in social service evaluation: A primer on agent-based modeling

Social Work Research

(2011)

Cited by (82)

How guest-host interactions affect consumer experiences in the sharing economy: New evidence from a configurational analysis based on consumer reviews
2022, Decision Support Systems
Citation Excerpt :
The nonlinear interaction of components implies that change in one component may insignificantly or hugely impact the entire system. Hence, understanding the system requires examining it as a whole, not its individual components [34]. The essence of complexity theory is its more in-depth explanation of how consumers evaluate services and outcomes [35].
This study examines the complexity of consumer experiences in the sharing economy (SE) from the perspective of the level of interaction between consumers and service providers. Consistent with service-dominant logic, the joint efforts of consumers and service providers co-create value. In the context of accommodation-sharing, this means not just the room that guests seek but, rather, the authentic local experience they co-create with their hosts. This study proposes a text-analytics framework to extract important service dimensions directly from consumer reviews. The results indicate that the importance of service dimensions, on which consumers focus in reviews, varies with levels of interaction. To better understand the complex nature of consumer experiences in the SE, the framework integrates text analytics with fuzzy-set Qualitative Comparative Analysis (fsQCA), to shift attention from individual service dimensions to service-dimension configurations. Drawing on complexity theory, this study examines the service-dimension configurations that lead to positive and negative sentiment. The fsQCA results reveal that the causal recipes for sentiment differ for various interaction mechanisms. This is the first study to integrate topic modeling, sentiment analysis, and fsQCA, framing service-provider decision support for responding to consumers' needs.
Evaluating the integration of strategic priorities within a complex research-for-development funding program
2021, Evaluation and Program Planning
This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.
The development and psychometric properties of an educational development impact questionnaire
2021, Studies in Educational Evaluation
Citation Excerpt :
Together these findings are important for understanding the types of services accessed by different instructor groups. To understand complex systems as a whole it is necessary to focus the evaluation approach across multiple levels over time (Walton, 2014). According to our outcomes framework, the common five factors map to four out of five long-term impacts (Fig. 5) which help us to consider the usefulness of the EDIQ for suggesting impact at the institutional level.
The purpose of this study was to develop and provide psychometric evidence of the Educational Development Impact Questionnaire (EDIQ) for evaluating complex outcomes of multifaceted educational development work within centres of teaching and learning. Our study addresses the need for a user-friendly and psychometrically sound survey instrument adaptable to differing higher education contexts. Our outcomes framework, mapping the intended short- and medium-term outcomes of our centre’s educational development activities, provided the necessary framework on which to create survey items. Scores from 267 instructors from a research-intensive university provided the data for examining the EDIQ’s initial factor structure using exploratory factor analysis. We justify our use of two different cut-scores to generate two models of factor structures. Factors common to both models are instructor growth, scholarship of teaching and learning, technology integration, impact on students, and knowledge enhancement. We discuss how the models provide different opportunities for assessing short-term and medium-term outcomes over time and the usefulness of the EDIQ beyond the current study context.
Evaluation of public health interventions from a complex systems perspective: A research methods review
2021, Social Science and Medicine
Citation Excerpt :
We consulted experts (n = 32) with an interest and expertise in systems-oriented public health research. We also conducted citation searches of relevant published systematic reviews beginning with two we had pre-identified (Carey et al., 2015; Walton, 2014). For our electronic search, we adapted search terms (Supporting Information 2) from those reviews and searched Scopus, MEDLINE and Web of Science, searching after the time period covered by the published systematic reviews (Carey et al., 2015; Walton, 2014).
Applying a complex systems perspective to public health evaluation may increase the relevance and strength of evidence to improve health and reduce health inequalities. In this review of methods, we aimed to: (i) classify and describe different complex systems methods in evaluation applied to public health; and (ii) examine the kinds of evaluative evidence generated by these different methods.
We adapted critical review methods to identify evaluations of public health interventions that used systems methods. We conducted expert consultation, searched electronic databases (Scopus, MEDLINE, Web of Science), and followed citations of relevant systematic reviews. Evaluations were included if they self-identified as using systems- or complexity-informed methods and if they evaluated existing or hypothetical public health interventions. Case studies were selected to illustrate different types of complex systems evaluation.
Seventy-four unique studies met our inclusion criteria. A framework was developed to map the included studies onto different stages of the evaluation process, which parallels the planning, delivery, assessment, and further delivery phases of the interventions they seek to inform; these stages include: 1) theorising; 2) prediction (simulation); 3) process evaluation; 4) impact evaluation; and 5) further prediction (simulation). Within this framework, we broadly categorised methodological approaches as mapping, modelling, network analysis and ‘system framing’ (the application of a complex systems perspective to a range of study designs). Studies frequently applied more than one type of systems method.
A range of complex systems methods can be utilised, adapted, or combined to produce different types of evaluative evidence. Further methodological innovation in systems evaluation may generate stronger evidence to improve health and reduce health inequalities in our complex world.
The Policy and Regulatory Engagement with Corruption: Insights from Complexity Theory
2022, European Journal of Risk Regulation
Post-project sustainability evaluation by using a complex systems approach: A framework and an application to a revolving loan fund (microfinance) project in Indonesia
2024, Systems Research and Behavioral Science

View all citing articles on Scopus

View full text

Applying complexity theory: A review to inform evaluation design

Highlights

Abstract

Section snippets

Methods

Results

Conclusion

Acknowledgements

Social Science and Medicine

Social Science and Medicine

Evaluation and Program Planning

Social Science and Medicine

European Journal of Operational Research

International Journal of Educational Development

Forest Policy and Economics

Social Science and Medicine

Social Science and Medicine

Evaluating health systems strengthening interventions in low-income and middle-income countries: Are we asking the right questions?

Health Policy and Planning

Case study research: The view from complexity science

Qualitative Health Research

Evidence understanding and complexity: Evaluation in non-linear systems

Evaluation

Wicked comparisons: Reflections on cross-national research about health inequalities in the UK

Evaluation

Using qualitative comparative analysis to understand complex policy problems

Evaluation

Selecting a change and evaluating its impact on the performance of a complex adaptive health care delivery system

Clinical Interventions in Aging

Developing an evaluation framework for crosscutting policy goals: The accessibility policy assessment tool

Environment and Planning A

Approaches to researching complexity in public management

Evaluating action learning: A critical realist complex network theory approach

Action Learning: Research and Practice

Evaluation in complex governance arenas: The potential of large system action research

Complexity configurations and cases

Theory Culture Society

Applying social science: The role of social research in politics, policy and practice

Evaluating complex social interventions in a complex world

Evaluation

Complexity theory and the social sciences: The state of the art

Evaluation and negotiated order: Developing the application of complexity theory

Evaluation

Complexity and life

Theory Culture Society

Complexity and whole-system change programmes

Journal of Health Services Research and Policy

The evaluation of health and social care partnerships: An analysis of approaches and synthesis for the future

Health and Social Care in the Community

Synthesising qualitative and quantitative evidence: A review of possible methods

Journal of Health Services Research and Policy

Dealing with complexity: Theory of change evaluation and the full service extended schools initiative

International Journal of Research and Method in Education

Applying complexity theory to New Zealand public policy: Principles for practice

Policy Quarterly

Food fight: The battle over redefining competitive foods

The Journal of School Health

Systems theory and complexity introduction

Democracy and Nature

Emergence as a construct: History and issues

Emergence

Knowledge theories can inform evaluation practice: What can a complexity lens add?

New Directions for Evaluation

Methods for exploring implementation variation and local context within a cluster randomised community intervention trial

Journal of Epidemiology and Community Health

Complexity theory and evaluation in public management

Public Management Review

Researching a local heroin market as a complex adaptive system

American Journal of Community Psychology

Nonlinearity in social service evaluation: A primer on agent-based modeling

Social Work Research