using an outcomes framework to design an indigenous youth program (case study 3 of 3)
This case study explains how I used an outcomes framework to help design an Indigenous youth program.
This is the third and final case study I am presenting this week.
Click here for an overview of the three case studies presented this week.
Happy reading!
Key points
- Using a framework that describes the possible immediate outcomes in the subject area of your program can be a valuable tool for program design.
- Going through this process with program staff who struggle with program logic is a great alternative to traditional program logic workshop approaches.
- Program staff responded well to a highly structured approach, which was presented in a series of simple steps.
I once worked as an internal evaluator for a youth –focused organisation – my role was to provide evaluation support to a range of programs across the organisation.
One of these programs was an Indigenous youth program. This program was comprised of a week-long intensive course. Young Indigenous people came from all over Australia to meet for a week and learn about being leaders in their community.
This was a relatively new program and was still finding its feet. We had made several attempts to create a program logic, but none of them had been completely successful in capturing what the program intended to do. I had pages of work on the ‘logic’ of the program, but it just wasn’t sitting together properly. Something was missing.
The program had run its first intensive the previous year. It didn’t go very well, and there was a lot of pressure on the team to improve the content of the course. The program had seconded a few staff from other programs to help write the intensive training content, but without a ‘logic’ to hang it on, it was proving difficult to design a cohesive week of lesson plans.
While all this was happening, I was also working on an organisation-wide outcomes framework. The outcomes framework was supposed to sit across all the programs I was supporting, helping us work out how all the programs when combined (including this Indigenous program) were contributing to the lives of young people.
The outcomes framework
The outcomes framework looked at a range of outcomes for young people. It was developed by synthesising qualitative data from interviews with approximately 50 program participants across a range of programs in the organisation.
This was a fairly unorthodox approach – the standard method is to read the existing literature and use this synthesis to construct a framework. But, while there was a lot of research examining outcomes for young people, I could not find any research that I believed was relevant to the organisation’s programs. I wanted to use evidence that captured the best of what the programs were doing.
At this early stage the framework only focused on immediate program outcomes. This was a deliberate choice –immediate outcomes are easier for programs to track and provide immediate information about whether the program is on track to achieving its long-term outcomes. Our idea was to validate the immediate outcomes and then turn our eye to longer-term outcomes by using a similar method of qualitative inquiry with participants who had completed the program three to five years ago. The outcomes framework would develop over time.
Below, you will find a copy of the framework (I hasten to add that at this point it was in draft form and still required some further development).
Click here to see a copy of the outcomes framework.
Because the framework focused on immediate outcomes, it proved to be an ideal tool for program design.
Designing the program
The program design methodology happened almost by accident. I was sitting at my desk and chatting with the two staff who were working on the program. I mentioned that I was working on the outcomes framework and was interested in knowing if it would be relevant to an Indigenous audience.
They said they would be happy to look at it. So although we sat down to examine the framework, we ended up designing the program!
Below are the steps we took that eventually helped determine the shape of the program:
Step 1 – look at the framework
The program staff and I sat down to examine the outcomes framework. Their feedback was very positive, but they wanted to add one extra outcome: ‘understand cultural knowledge’. This outcome was added to the ‘values’ stream.
Step 2 – circle the outcomes you want to your program to contribute to
I then asked the staff to circle all of the outcomes that they were hoping their program would be able to contribute to. They circled around 20 outcomes.
Step 3 – refine these further to 8–10 of the most important outcomes
20 outcomes was too many for the process. The staff were able to bring it down to 8 of the most important outcomes.
Step 4 – get staff to describe the journey to the outcome
I then asked staff, in relation to each outcome:
- Where do you expect the participant to be at the end of the week?
- Where do you expect the participant to be at the beginning of the week?
- How do you expect the participant to get from the beginning of the week to the end of the week?
We listed the eight outcomes on the wall like this:
Where they are at the beginning of the weekWhat happens during the weekWhere they are at the end of the week
The outcome as identified in the framework.
A statement that describes where the participant is at the end of the week.A statement of what happens to get from BEGINNING to OUTCOME. (These are used in the next step)Where the participant is at the end of the week.
EXAMPLE
Unable to articulate values of their own community.
EXAMPLE
Activities to uncover their community values
Activities to get participants to talk about their communities with each other
EXAMPLE
Better understanding of the link between their values and communities values
Understand that different communities have difference values
Able to articulate values of their own community
They don’t know the difference between their values and the values of their community.
Activities that get participants to understand what their values are.
Activities that get participants to analyse where their values are similar and different to those in the group.
Activities to get participants to analyse the difference and similarities between their values and their communities values
Understand the difference and similarity between their values, the values of others, and the values of the group.Able to identify personal values
Step 4 – ordering the BEGINNING to OUTCOME statements from the previous step in sequential order
Once we had done this for each of the outcomes, we arranged the descriptions of the journey from BEGINNING to OUTCOME in a time-based order. This was based on the assumption that participants need to learn things in a particular order, like building blocks being stacked on top of each other.
I asked the program staff which of the statements in the second column of the table above would need to happen first. I then asked which one would need to happen second, third and so on.
The statements were rearranged on the wall in this order:
This happens firstThis happens secondThis happens thirdThis happens fourth (and so on...)
A statement of what happens to get from BEGINNING to OUTCOME.A statement of what happens to get from BEGINNING to OUTCOME.A statement of what happens to get from BEGINNING to OUTCOME.A statement of what happens to get from BEGINNING to OUTCOME.
EXAMPLE (taken from column two previous step)
Activities to uncover their community values
Activities to get participants to talk about their communities with each other
Activities that get participants to understand what their values are.
Activities that get participants to analyse where their values are similar and different to those in the group.
Activities to get participants to analyse the difference and similarities between their values and their communities values
And so on…
This framework was then used to help the facilitators design the lesson plans for the week.
As a result of completing this activity, I was able to plan the program logic. The logic is a little bit different to a usual logic, as it includes the learning journey developed through this process and connects it to the expected immediate and longer-term outcomes. A program logic is generally built by identifying the broad goals and working backwards, but we got a lot more traction by using this immediate outcomes method.
Click here to see a copy of the program logic.
The feedback I received was that the second intensive was a significant improvement on the previous year. There were many reasons for that, but the planning process contributed to building a high-quality program. Program staff said that they found the process easy to follow and as well as culturally sensitive.