The complexity of citizen experience: 'System Effects' mapping for intervention design

System Effects is a methodology developed by UNSW Canberra Researcher Dr. Luke Craven to explore the ‘user’ or citizen experience of complex phenomena, such as climate resilience, poor health, or job market access. The method is proving to be useful for citizen and user engagement worldwide, and Luke details it's varied applications and processes for us here. 

The System Effects methodology emphasises the varied nature of social phenomena, their causes and consequences, while at the same time giving policymakers tools to understand the complex nature of how those varied factors manifest at the community- or population- level. System Effects can be used to support the design, implementation and evaluation of interventions aimed at changing the structure of complex adaptive systems to drive particular outcomes. By beginning from the ‘user’ understanding of complex systems, the methodology helps to re-centre lived experience in social science and policymaking practice.

UNSW has produced a great video summary of the method, and Luke writes about it in detail for Power to Persuade below:

Developed as part of my PhD that focused on developing new tools to understand and address food insecurity from a systems-based perspective, System Effects is increasingly being applied to a whole range of issues by national, state, and local governments across the world. For example it is being used to:

1. understand the barriers to job market entry in Oslo, in partnership with the Norwegian Labour and Welfare Administration (NAV);
2. understand the systemic impact of disaster events in Sydney, in partnership with Resilient Sydney and the NSW Office of Emergency Management;
3. support social workers to deliver systemic care to persons facing homelessness in Newcastle, UK, in partnership with Newcastle City Council;
4. support the development of policy to prevent food borne disease in Cambodia, in partnership with the International Livestock Research Institute (ILRI) and USAID, and;
5. support effective environmental stewardship in New York, in partnership with the US Forest Service.

The above is just a snapshot of the diversity of issues and decision contexts in which System Effects is currently used, with a more detailed picture in the map below.

Figure: Where is System Effects being used?

But what exactly is System Effects and how does it work? The methodology draws on soft systems methodology, fuzzy cognitive mapping, and graph theoretical analysis. Its objective is to aggregate and quantify participant-generated system models of a given problem (e.g. poor health or malnutrition) and its determinants to inform intervention design.

The participant-led approach begins by asking research participants to visually map or depict the range of variables they perceive be causes of the problem at hand; drawing arrows between the variables to indicate causality.

Once completed, the researcher creates an adjacency matrix for each participant response, using a coding scheme to ensure consistency in the factors present across the community. The foundation of this method is that individual participant maps represent network diagrams, with the barriers between them acting as ‘nodes’ and the connections between them as ‘edges’ or links. As an example, a truncated participant map from my work understanding the primary determinants of food insecurity;

Will result in the matrix below:

Next, these individual adjacency matrices are aggregated into one matrix that brings together individual perceptions. The aggregate weight of connections between variables depends on the frequency with which that connection is identified across the sample. Aggregating individual mental models in this way accounts for every variable identified across the sample, and the diverse range of causal pathways present, as well as making clear the intensity and frequency of particular connections at the community- or population- level.

This aggregate adjacency matrix can be used to create a directed network graph, where the nodes represent barriers to food access and each connection represents the number of participants identifying particular links between them. This is done using the network mapping software Gephi. An example from my work on food insecurity can be seen below.

Finally, these aggregate models can then be explored using graphic theoretical analysis to identify key structural components of the system. Unlike many analytical approaches that emphasize linear associations, System Effects emphasizes the complexity and heterogeneity of participant responses. These statistics provide a way to ‘navigate’ the complexity of a given problem and its determinants in aggregate at the community- or population- level, providing evidence to assist in the design, implementation and evaluation of interventions to address it.

The method presents opportunities to identify key structural features of systemic problems across communities, providing insights into their structure as complex systems.

The results can be used to inform policy and program:

• Design: How can policies most effectively address systemic and complex determinants of disadvantage, given its heterogeneous nature?
• Implementation: How can the implementation of policies be effectively tailored to the systemic dynamics of particular contexts?
• Evaluation: How can we assess the systemic impact of particular interventions and their interactions with the contexts in which they are deployed?

The System Effects approach can and is being adapted to help answer each of these questions providing a range of tools to assist policymakers in promoting systems change in disadvantaged communities. More information about System Effects is available here, and for those lucky enough to be in Newcastle in June, do think about coming along to this free(!) workshop on the methodology, its popularity and potential.