The evaluation theory tree

I have been working in the field of evaluation for the past eight years and I think what I have learnt most about evaluation in this time is that it is a crowded market. A lot of people are involved in the practice of evaluation. And while I have never conducted a scientific study, I have found it truly shocking how many practising evaluators know little to nothing about evaluation theory. Or know a lot about applied research methods and consider this an acceptable substitute for knowledge about evaluation.

Over the past few years I have been trying to ‘fight the good fight’ and disseminate information about evaluation knowledge and theory. I have had many evaluation positions within organisations and if I were going to be metaphorical about it I would say that I was like a missionary preaching the good word and trying to convert people. The problem sometimes with missionaries though is that they lend themselves to opaque and obscure posturing, and they sometimes assume that their truths are self-evident.

Still I continue to encourage people who are touched by evaluation to learn about how some of the ‘big gun’ evaluators think and why they think it. This is important for a few reasons. Firstly, it will make your evaluations better. Second, a good evaluation will help you to better serve the people your programs say they help, and thirdly, it will give you ammunition next time you need to fight an evaluator.

 As the resident evaluator at Power to Persuade I am going to be disseminating more information about evaluation theory and practice on the Power to Persuade blog. Today, I start with a précis of a book chapter - which has been enormously influential in my evaluation practice. The chapter is called ‘The Evaluation Theory Tree’ and is from the book Evaluation Roots, written by Marvin Alkin and Christina Christie. It was first published in 2004 by Sage.

 What is in the tree?

In the evaluation theory tree Alkin and Christie construct a taxonomy of evaluation theory. The tree has two trunks/roots and three branches.

Figure one: the evaluation theory tree, as found in the Alkin/Christie book, Evaluation Roots.

The two trunks/roots are:

• Accountability and control – the rationale for evaluation
• Social Inquiry – the methods we use to collect the information we need to determine accountability

The three branches are:

• Use – how to make sure that evaluation findings are used by the intended audience
• Methods – concerned with the use of social science research to make statements about reality
• Valuing – this is what distinguishes evaluation from research. Evaluators must place value on their findings

I really like this chapter, as I believe it provides a succinct account of the theoretical traditions of evaluation. Understanding these theories will really help you to make good decisions about your evaluation practice, to think about what you need to know, why you need to know it, and who the evaluation findings should be used by.

But how do I know what I know?

I have come to believe that there is a deeper root missing from this tree, sitting underneath the ‘Accountability and Control’ and ‘Social Inquiry’ trunks. Modern evaluation practice is influenced by a range of theoretical traditions to do with the way in which knowledge is constructed and the reasons why we construct this knowledge. This deep root is called epistemology.

Last October I published what was intended to be a humorous quiz, which was called ‘What Type of Program Evaluation Are You?’ It was a short five-question quiz and it was designed to engage people in evaluation in a relatively light-hearted way.

Here is a link to the quiz: http://powertopersuade.org.au/2014/10/31/friday-fun-what-type-of-program-evaluation-are-you-take-this-easy-quiz-to-find-out/

However, whilst the quiz is light-hearted, it is actually designed to help you think about where you might fall on a spectrum of preferred types of knowledge construction as they pertain to evaluation practice.

Epistemology is the enquiry into what can be known, and how we can know it. It is concerned with the difference between our knowledge and our beliefs, and investigating ways in which we can reliably make distinctions between what we know and what we believe. You could spend an entire postgraduate degree purely devoted to investigating epistemology, it is a complex subject and there is no way I am going to be able to do justice to the subject matter in this article.

There are a range of differing epistemologies that are used in evaluation practice and they are all empirical. That is, they are all based on the observation of the world; evaluation is an assertion of physical fact. The practice of evaluation is concerned with the best methods for revealing these facts and then how to use these facts.

Evaluation epistemology tends to run across a spectrum from scientific to constructivist.

• Scientific knowledge posits that there is a reality that lives outside of our minds. We use falsifiable methods for describing this reality. Knowledge is constructed through objective observation of reality.
• Constructivist knowledge posits that there is a reality but it is mediated through society and culture. Knowledge is constructed through subjective observation of reality.

Below I have included a copy of all five possible ‘What Type of Program Evaluation Are You?’ quiz results and where the fit on an epistemological spectrum: from scientific to constructivist.

So now I turn to describing the evaluation theory tree. The tree has two trunks and three branches. The two trunks are: accountability and control and social inquiry. The three branches are: use, methods, and valuing.

 TRUNKS

Alkin and Christie proposed that there are two ‘trunks’ that underpin modern evaluation practice.

Accountability and control

This is the rationale for evaluation. If we are running a program that is intended to address a social problem, then we have a responsibility to be accountable for the outcomes of the program.

The theorist Wagner said that there were two phases of accountability: description and explanation

The first phase is reporting. This is about providing a description of what happened.

• Process accountability is about determining whether ‘reasonable and appropriate procedures for accomplishing goals have been established and implemented’.
• Outcome accountability: the extent to which the intended goals have been achieved

The second phase of accountability is an explanation of what happened and holding those who were responsible for the program accountable.

Evaluation provides the information we need to hold organisations to account for their programs.

Social inquiry

Alkin and Christie write that “the social inquiry root of the tree emanates from a concern for employing a systematic and justifiable set of methods for determining accountability.”

Social inquiry is the study of groups of individuals in social settings. It wasn’t until the 19th century that society and social groups began to be studied empirically (i.e. study of groups based on observation), when people such as Karl Marx, Emile Durkheim and Max Weber began collecting data on social groups and their interaction and using statistical analysis to make statements about the nature of social groups.

There are ongoing debates in the field of social science as to the best methods for the study of social groups and whether this study should incorporate the methodologies of the physical sciences. (Remember earlier we discussed the differing epistemological approaches to evaluation, with objective scientific inquiry on one side of the spectrum and subjective constructivist enquiry on the other side. The debates in evaluation reflect these epistemological debates.)

There are two broad types of social inquiry: explanation and prediction, and interpretation and understanding.

• Explanation and prediction forms of inquiry are more aligned with scientific research methods such as experimental design and statistical analysis. This is about using your research to make generalizable statements about interventions. Generalizability is about the extent to which you can apply your research findings to the broader population.
• Interpretation and understanding forms of inquiry are more aligned with constructivist research methods and qualitative inquiry. This is about using your research to understand why things happen and what it means.

BRANCHES

The branches on the tree are methods, use, and valuing.

The three branches are:

• Methods – concerned with the use of social science research to make statements about reality
• Valuing – this is what distinguishes evaluation from research. Evaluators must place value on their findings
• Use – how to make sure that evaluation findings are used by the intended audience

Methods

This branch is related to explanation and prediction. The social sciences have often used scientific methods found in the physical sciences. The emphasis on this branch is primarily concerned with the conduct of scientific inquiry and in particular on the use of well designed experimental studies and quasi-experimental studies to be able to make generalizable statements about programs and their effects.

Some evaluation theorists consider the quality of research methods as being of the greatest importance in evaluation because it increases the power and influence of the findings generated by the research.

Experimental design

Experimental design in evaluation is seen as being critical to making statements about which programs have caused which changes in certain types of social groups. Randomisation is considered essential to well designed experimental methods, where subjects are randomly assigned to treatments, because ‘random assignment ensures that experimental and control groups are, on average, equivalent before the treatments were introduced’. This allows researchers to be able to make generalizable statements about the research results.

Because pure experimental design is not always possible (or ethical) in field research, other evaluation theorists began advocating for quasi-experimental design. This is similar to experimental design in that you have a treatment group and a control group. The key difference is that subjects are not randomly assigned to treatment groups. To some theorists this weakens the capacity of the evaluators to be able to make causal statements about the treatment and its effect. Other theorists argue that it is not feasible to implement a pure randomised experimental model to investigate social programs and the quasi-experimental design is seen as a compromise.

Objectives-oriented evaluation

This term was coined by Ralph Tyler, an American educator. The objectives-oriented approach focuses on a) formulating a set of objectives/outcomes/goals (‘the theory’) b) classifying these objectives into major types and c) selecting methods for obtaining evidence regarding each objective d) devising means for interpreting results.

Objectives-oriented approaches to evaluation were the genesis of the use of theory of change, or program logic, in evaluation. (I will use the phrase program logic) The purpose of the program logic is to provide a theory about the program that is then used to guide the evaluation methods. This theory is very important because it helps people to understand where the program worked well, where it did not work well, and what parts of the theory need improvement, or alternatively which parts of the program need improvement. It you want to make generalizable statements about a program, you can then look at where programs adhered closely to the program logic and where the program diverged. In theory you should be able to implement the program again in a different setting using the program logic that has previously been tested in an evaluation.

Valuing

This branch is about what makes evaluation distinct from pure research: the role of the evaluator in placing value on their research findings.

The evaluation theory around placing value runs along a continuum of valuing:

It is the role of the evaluator to make evaluative statementsThe evaluator is the mediator and reporter of a range of evaluative statements and perspectivesThe evaluator provides information and assists other people to provide evaluative statementsEvaluation in pursuit of social justice

The evaluator as the person who makes value statements about the program

 Particular theorists maintain that it is the role of the evaluator to make ‘evaluative statements’ about the program they are investigating. The most famous proponent of this branch of theory was Michael Scriven. He said ‘there is a science of valuing and that science is evaluation’. Another theorist, Eisner, talked about the evaluator as having ‘the enlightened eye’ - a good evaluator needs to have the skills to be able to value and judge a program. These theorists both see the evaluator as having a special set of skills that makes them uniquely suited to the task of making evaluative statements about programs.

Evaluator as mediator and negotiator of a range of evaluative statements

Various theorists position the evaluator as the negotiator of multiple realities and perspectives. For example, evaluations will often have multiple audiences. There will be the people in power (Government, program staff), and interested community stakeholders (who will have a range of realities and perspectives). These audiences may have competing or aligned interests in the evaluation, and it the role of the evaluator to mediate these interests and represent the valuing statements made by all interested parties. For this reason the evaluator must seek to identify as many interested stakeholders as possible to ensure that the maximum number of perspectives are accounted for.

The issue of ‘multiple realties’ is particularly important in evaluation. Remember at the beginning of these notes when we talked about epistemological approaches. Different groups will have differing perspectives on what constitutes knowledge. A good evaluator is aware of these issues and able to negotiate them.

Evaluator as provider of information to other parties

Some evaluation theory focuses on the evaluator as the provider of information for others so that they can make evaluative statements. In this branch of theory it is the role of the evaluator to act as expert advisor and technical expert, and provide the information necessary to assist others to make judgements about a program.

Evaluation in pursuit of social justice

This branch of evaluation theory is concerned with representing traditionally marginalised groups in the evaluation process. Some theorists position evaluation as an important tool for social justice.

A particularly well-known proponent on this branch is David Fetterman. Fetterman coined the term ‘Empowerment Evaluation.’ This is a form of evaluation whereby people who are the intended beneficiaries of an evaluation are tasked with carrying out the evaluation and making evaluative statements. The intention is that program beneficiaries are empowered through this process and have greater control over what programs do and who they benefit.

Use

This branch of evaluation is concerned with the conduct of evaluations that assist key program stakeholders in program decision-making. This is because the likelihood of evaluation being used is greater if the people who have a stake in an evaluation are involved in the development of the evaluation and its findings.

This branch focuses on:

• Identifying people with the greatest stake in the evaluation
• Methods that will increase uptake of evaluation findings
• Providing information to decision makers to enable program improvement

Identifying people with the greatest stake in the evaluation

Beliefs about who has the greatest stake in the evaluation and who should be targeted differs from theorist to theorist, but essentially covers: people who deliver a program, people who make decisions about programs, and expected beneficiaries of programs.

Methods that will increase uptake of evaluation findings

In the Use branch evaluation theorists consider it crucial that evaluation processes engage a range of stakeholders throughout the evaluation process to ensure buy-in and develop evaluation findings that will get used by the intended audiences. Patton believes that the evaluator has an obligation to ensure that utilisation of evaluation findings takes place.

Providing information to decision makers to enable program improvement

Evaluation theorists on this branch believe that evaluations should provide streams of information (through monitoring and reporting) to help programs to continually improve.

The most well known theorist on this branch of the tree is Michael Patton. He coined the term ‘Developmental Evaluation’. In this type of evaluation, the evaluator becomes part of the team and fully participates in organisational decision-making and facilitates discussion about evaluative findings to increase uptake.

- Lauren Siegmann