Impact or outcome, what's the difference??

We've run a few pieces this year on the impact/outcome funding debate. James Rees has run a three part piece (part 3 still to come) on different aspects of the impact agenda, including possible pitfalls and how one might differentiate the contribution of community-based organisations within such an agenda. Fiona Buick, Pauline McLoughlin and I also wrote piece on the potential benefits of outcomes based funding. Impact and outcome tend to get a little tangled up in these discussions. Below, Andrew Harding does a little disentangling. Andrew Harding is undertaking a doctoral research project at Bournemouth University with Elderly Accommodation Counsel (EAC). This piece first appeared on the LSE Impact Blog.

In recent years ‘impact’ and ‘outcomes’ have become familiar terms to academic researchers. However, the differences and nuances of ‘impact’ and ‘outcomes’ are often rarely discussed and possibly poorly understood by academic researchers. In this blog post I want to draw upon one research base in particular, reflect on the terminology of ‘impact’ and ‘outcome’, outline what I regard to be the key differences and even go as far as proposing that the distinctive nature of each aligns itself to opposing research methodologies.

The field I refer to is the impact and or outcome of information and advice on welfare. It is fair to say it is an established, but remarkably under-nourished field. However, before I discuss some examples of research in this area, and how methodologies are used when reporting on impact or outcome, it is best to outline the definitions of impact and outcome that I feel are valid. I will then propose that it is possible to align opposing research methodologies to each. Existing research will then be reviewed in the context of these distinctions.

The key differences between ‘impact’ and ‘outcomes’: reach, scope and nature

In the field I am referring to, the key distinctions between ‘impact’ and ‘outcome’ can be illustrated in the different reach, scope and nature of each. First, take outcomes. An outcome is a finite and often measurable change. On this basis, the reach of an outcome will be pre-defined and the scope of an outcome will be similarly limited. For example, an outcome of an information and advice intervention on healthy eating, nutrition or weight loss might be changes in bodyweight or body fat etc. Another example of an outcome, this time of an health related information and advice telephone service, may be whether someone has sought out and accessed further health related support in their community. In these examples, outcomes are focused on the measurable objective changes that are brought about by engagements with information and advice. For example, an individual lost 5 stone or 15% of his/her body fat after receiving a form of information and advice on healthy eating, nutrition or weight loss. Similarly, an individual would have or would not have sought extra support in his or her community.

Image credit: Clickpath Analysis by KTB (Wikimedia, CC BY 3.0)

In contrast, impact refers to a much broader effect – perhaps the effect information and advice had on ability to make an informed choice, empowerment or wider life experiences. Impact can be conceptualised as the longer term effect of an outcome. For example, to take the examples uses above, the impact of losing a lot of weight or body fat might be an increased sense of happiness and or a decreased sense of insecurity. The impact of accessing local community support might be an increased feeling of peace of mind and or emotional support. When compared to outcomes that tend to be pre-defined and can be measured objectively, the personal experiences and inherently personal nature of impact is intuitively subjective.

There is evidence to suggest that these distinctions are supported by organisations like the Charities Evaluation Services and the Centre for Nonprofit Management. The latter state that “While the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants’ lives on a broader scale.”

Aligning impact and outcomes to opposing research methodologies

Although the distinctive nature of each is outlined here, the interlinked and sequential nature the relationship between impact and outcome is also apparent. This will almost certainly be of interest to those tasked with designing social research. When interested in objective outcomes, it is appropriate and there is an established tendency to use quantitative methods. On the other hand, impact is sufficiently different, complex, non-linear and ultimately subjective to warrant a qualitative approach. However, it is not as straight forward to only indicate that outcomes support quantitative methods and impact lends itself to the strengths of qualitative methods. There are other appropriate methodological configurations. If consideration is given to the interlinked and sequential relationship between outcomes and impact (and vice versa), a mixed methods approach that focuses on delineating outcomes and exploring impact might be appropriate. Or, if a detailed exploration of agency – or how individual use of information and or advice shapes the processes around outcomes and impact then an in-depth qualitative case study approach is likely to be appropriate.

A critique of existing research on the impact and or outcomes of information and advice on welfare

With these methodological definitions in mind, how is existing research approached and designed? As part of my doctoral research, I have reviewed studies focusing on instrumentally rational participants – i.e. studies of information and advice services where individuals initiate contact. This criterion tends to exclude a large number of studies where the focus is on health or medical information/advice that is imparted based on a professional assessment of need. On this basis, what is said here is based on a sub-section of the impact/outcomes of welfare information and advice literature. A detailed account of the inclusion/exclusion criteria and search approach is for elsewhere.

There is a dominant preference for the use of quantitative methods when reporting outcomes. As outlined above, on the face of it this is methodologically sound. However, a closer look at this body of work reveals some significant deficiencies in this area. In particular, two main and broad critiques that focus on research approaches and terminology are evident.

Firstly, a preferential pursuit of quantitative outcome measures does not constitute an expansive and comprehensive research base. There is a relative paucity of studies that focus on impact as a singular focus, or as a complementary element to studies that report outcomes. This severely limits the scope of research findings. For example, one study finds that a good percentage of participants accessed local community support – but nothing is reported on the more subjective and finer details of the broader impact this outcome had on participants’ lives. A tendency to report outcomes, and not impact in adequate detail, is a common feature of the sub-section of this literature. In fact, one paper acknowledges this as a weakness of a quantitative research design – calling for the use of qualitative methods to compliment quantitative findings in order to provide detail on impact.

Secondly, there is also a great deal of variation in the use of terminology and the assumptions about the nature of what an impact or outcome refers to. These are assumptions because, despite impact and outcomes being central to these studies, the nature of each is not discussed at all. For example, one study claims to report on the ‘impact’ and ‘experiences’ of participants. These are aspects which as outlined above are intuitively personal, subjective and best captured by qualitative means. However, the study reports quantitative measures – a large number of which appear to be performance metrics on areas such as approachability and clarity of advisors. Where ‘impact’ is the focus of data collection, for example on anxiety, quantitative measures are used – despite the uneasiness and critiques discussed here and by other researchers.

In this blog I have outlined that there is some confusion around the definitions of ‘impact’ and ‘outcome’, which is possibly also reflected in the how studies are approached, the use of terminology and the use of research methodologies. I do not claim, in such a short blog post, to have offered a comprehensive and definitive ‘solution’ to these issues. Rather, what is clear is that this area would benefit from some serious discussion – of which I hope this can make some sort of contribution.

Andrew Harding is undertaking a doctoral research project at Bournemouth University with Elderly Accommodation Counsel (EAC). Based in London, EAC coordinates the FirstStop service which provides information and advice to older people (and other stakeholders) on housing and care issues. My research is focused on how older people use information and advice and the wider impact of information and advice on housing.