Research Engagement and Impact: The rhetoric, the evidence, and the practice
The Australian Research Council (ARC) has been charged with assessing research engagement and impact and Australian universities have clambered to produce evidence of it. The Australian Engagement and Impact Assessment pilot is complete and everyone is getting ready for it as part of ERA 2018. So, what is it?
The ARC have developed the following definitions:
· Research impact: the contribution that research makes to the economy, society and environment, beyond the contribution to academic research.
· Research engagement: the interaction between researchers and research end-users (including industry, government, non-governmental organisations, communities and community organisations), for the mutually beneficial exchange of knowledge, technologies and methods, and resources in a context of partnership and reciprocity.
The Engagement and Impact Assessment is focused on measuring engagement and assessing impact. The ARC website indicates that the pilot included engagement metrics and case studies of impact. Further details are yet to be publicly revealed.
The rhetoric, however, reveals a central assumption: increasing engagement is expected to increase impact. The premise is thus: if we increase engagement with industry, government and other decision-makers then we increase the likelihood that engagement will result in research being used in some way that contributes to economic, social, environmental and other benefits. If you take a quick look at the body of research on research translation and implementation science (fields of research specifically focused on how to get research used in policy, programs and practice) this assumption appears sound. Many factors found to affect research use are related to engagement. Collaboration, relationships with decision-makers, and improved dissemination were amongst the top five enablers identified in a 2014 systematic review of factors affecting decision-makers use of research.
The challenges inherent in this assumption become apparent when you dig a little deeper into the evidence base. The review authors have shown that the majority of existing research on factors affecting research use is based on perceptions; surveys and interviews on what decision-makers and researchers think affects research use. When examining reviews of studies that have tested assumptions about how to increase research use, the picture becomes much more complex.
A 2016 review of systematic reviews identified 16 high quality studies that specifically tested approaches to increasing decision-makers use of research. Nine of these studies failed to increase research use and seven showed an improvement in use of research. This highlights that quite often, our assumptions about what will increase decision-makers or end-users use of research are not borne out in practice.
What we do know is that a complex interplay of factors must work in alignment to drive research use. For example, there was reliable evidence that increasing access through communication strategies or repositories did increase research use, but only if the intervention simultaneously sought to increase decision-makers motivation and opportunity to use that research. The review authors concluded that ‘… the intention to use evidence, in itself, cannot be regarded as a reliable indicator...’ of research use.
At present the evidence base points to ‘best bets’ about what might work to increase research use. Even then, this comprehensive systematic review has shown that the evidence we do have ‘tells us little about how these interventions work’ (emphasis added). This highlights that a strategic evidence-based approach to research engagement, translation and implementation is needed to support and enable and the use research that is necessary for research to have an impact.
The good news is the ARC is going to reward us for engagement even if it doesn’t lead to impact. Engagement has been coupled with impact in Australia; recognising the fact that researchers are not in direct control of whether decision-makers use the research they have funded. The risk with this approach is in incentivising activity that is not directly linked to the intended outcome. By increasing engagement without clear evidence of strategies are that effective in increasing decision-makers use of research, we risk increasing, or at least sustaining, industry and government perceptions that academics fail to understand decision-maker needs (for e.g. see Miles review of Collaborative Research Centres and a recent US study).
This is not to suggest we should not move forward with engagement and impact activity. The Engagement and Impact Assessment has provided the critical incentives needed to support investment in engagement and impact work. But unless we take our own advice and use research evidence to inform engagement and the collection, analyses and implementation of the impact data we will now collect, then we will miss a great opportunity to learn how to overcome the significant challenges faced in seeking to drive industry and government use of research.
We need to be collecting detailed data and evidence throughout the engagement to impact pathway to increase our understanding of how to achieve impact, and how this varies across different types of research projects and decision-maker and end-user contexts. This will bolster the evidence base needed to help researchers and institutions take a more strategic approach to research engagement, translation and impact planning, monitoring and evaluation. If we can use this opportunity to better understand how to increase research use, then we can significantly increase the chance of our research achieving real and lasting impacts on the world beyond academia.
Dr Pauline Zardo currently works as a Data and Post Doctoral Policy Fellow at QUT undertaking research in the area of open access and data driven impact. Dr Zardo completed a PhD focused on increasing government decision-makers use of research at Monash University in 2013. Since then she has worked in various academic and professional roles in the higher education sector, including as Research Translation Manager at Monash University and Research Impact Officer at UQ. Prior to undertaking a PhD Pauline worked as Strategic Policy and Project Officer in State Government in Victoria.