The best policy decisions need the best data, which means we need to work together

2020 was the year where we had to consistently ‘pivot’ – our workstations, programs, research agendas, and funding. At a social level, we saw Federal and State governments, not-for-profits, and peak advocacy bodies make decisions about who or what were policy and funding priorities off the backs of bushfires, a pandemic, and a recession.

Where policy attention gets directed is influenced by data in a myriad of forms – including census estimates, one-off surveys, and administrative data. Also, and perhaps more influentially, are the qualitative accounts and anecdotes fed back to elected officials and philanthropists or other funders about overrun services or equipment shortages. With so many issues jostling for priority, having data that is reliable and easy to access and interpret is vital for both policy makers and those running organisations. This post is by this week’s moderator, Dr Megan Weier, and Centre for Social Impact Research Assistant, Isabella Saunders.

Image credit: NASA

As the lead researchers of the Australian Social Progress Index (SPI), we have spent two years becoming acquainted with the myriad forms of data that is collected, covering the economic, social and environmental in Australia. Scoping out what data is available, and how often it is updated highlighted some immediate priorities for how data is currently being collected and disseminated. Meeting and talking through the Index with members of parliament, public service, and the not-for-profit sector also revealed that there are both some known and unknown realities about the current quantitative data environment both within states and territories, and at a national level.

Without access to up-to-date, consistent, reliable and well-defined data, there is a risk that social issues are misunderstood. Further, we cannot track the impact (positive or negative) of a policy change or program initiative – and we instead rely heavily on anecdotal data, or try to measure outcomes ad-hoc. But as this last year has shown, social, economic and environmental issues do not suddenly appear when there is a crisis – so having data collection and dissemination processes already in place should be a long-standing priority.

Here, we outline what we believe are the greatest priorities for social data in Australia. These are issues that require multiple stakeholders to be aware of, and a willingness to partner in improving how data can be used in determining joined-up and integrated policy and programs.

Know what is out there

Knowing where to look for data is a specialised skill in itself. Federally-funded research centres, such as the Australian Bureau of Statistics (ABS), the Australian Institute of Health and Welfare (AIHW), and the Australian Institute of Family Studies (AIFS) form major repositories of informative data. In addition, annual surveys such as the Melbourne Institute’s Household Income and Labour Dynamics in Australia (HILDA) survey and Productivity Commission’s Report on Government Services provide additional social insights that can complement administrative data. However, each organisation has its own timelines for publishing data, and the degree of analysis and interpretation can vary depending on both the dataset and the organisation.

A key challenge here is knowing what data sources are available, including its geography (i.e. is it reported at a national level, or could it be reported down at a community level like the Census?), how often it is updated, what analyses are available, and how to access the data. An ideal solution is a single, comprehensive directory that can be used as a starting point when looking for particular data. The ABS has a publication release calendar that outlines the frequency of updates and direct links to each of the publications and their associated data. Having this information for the most commonly used data sources in one location could help ensure the right data is referred to, and help avoid replication of similar datasets.

Up-to-date decisions need up-to-date data

Not all data can be collected every year – we would all get sick of filling out the census every year (not to mention the cost!). But even data that is updated annually can have significant lags between when it is collected and when it becomes available. In the process of sourcing data for the Australian SPI, we found that most social data only become available 10-18 months after it was first collected. Appropriate cleaning and management is required of large datasets, however we must be cautious of using social data that reflects potentially vastly different social contexts between when it was first collected, and when it is used to inform policy or practice. This may require a revisit of, for example, the process of reporting data first to state or territory agencies, which then get passed on to a national agency for analysis and reporting. If there are solutions that can ease reporting burden and also potentially reduce the risk of misreported data, everybody wins.

When trying to understand social issues, we usually want data that can provide historical context. For this reason it is important to prioritise continual data collection, with minimal changes to the questions being asked or the way the data is collected. Aiming to have a continual replication across most, if not all, social data will provide the most accurate picture of change.

Focus on outcomes, not just inputs

When searching for indicators to include in the Social Progress Index, a requirement was that the data used were measures of outcomes as opposed to either inputs or outputs. Inputs give information about resourcing (e.g. amount of money spent on social housing programs or proportion of children fully vaccinated by age 5), but they don’t necessarily tell us whether an initiative worked, or led to the result a policy or program expected to see. Likewise, outputs – such as number of vaccines delivered or number of reports that are released after an inquiry, give us as sense of ‘things’ that happened in relation to a particular event, but cannot provide feedback about the value or impact of a service or policy.

Outcomes, however, can be used to understand the lived effects as a result of a policy or program. Most outcomes are not attributable to a single initiative, but can be the most direct way to understand progress and change. For example, a decline in the estimated homelessness rate is a useful outcome indicator that housing and homelessness approaches are working effectively. Some input and output data in inevitable, but in order to make the most data-informed decisions, we need to be clear about what an indicator is measuring, to preference outcomes-based data collection, and to have this data measurement plan in place as early as possible.

Data is political

Numbers can often be treated as neutral or a-political reflections of ‘truth’. However, the ways data is collected – and even what data is not collected – how it is reported, and how it is used in broader social narrative is inherently political. An example of this comes from our search for environmental quality data that is collected consistently across states and territories. We mostly relied on air quality index reports, as there were no consistent and recent reports on other environmental indicators that are reported below a national summary, such as greenhouse gas emissions or tree clearing. This lack of data makes it difficult to mobilise calls for greater action on climate change – and it can also make it easier to deny that there are social or environmental issues if there is no data there to back it up.

The ‘who’ of who is being measured also requires thought. Groups who are most affected by social issues are often not measured. This was a limitation that we found, when, for example, trying to find data that reported experiences of racism or discrimination. This requires specific data that is informed by minority groups, rather than an umbrella measure across an entire sample.

Data cannot ‘tell’ us anything on its own, but is instead guided by our own interpretations, past knowledge, and is often interpreted in relation to other data points and ideas. This is not necessarily a bad thing, but it does require that for those of us who collect or interact with data to employ greater reflexivity about our own roles in the data analysis and interpretation processes.

Work with each other

Through the processes of evaluating more than 450 social and environmental indicators, it was enlightening for us to see how much data is collected across the country. There were many indicators that were well measured or useful, but because of low sample size, once-off collection, or it was only collected in one state or territory, it could not be included in the Index. What we saw was that there could be a number of similar surveys or research projects, measuring indicators in slightly different ways. A robust and sustainable approach to long-term data collection requires partnership and collaboration. By working together (such as using previously used and validated indicators, partnering in related research projects), we can aim to collect data that results in better sample sizes and reflects better granularity across the country. A ‘collect once, use often’ approach can ensure that we are not collecting data simply for collections sake, and, amongst other benefits, reduce research fatigue.

How data is collected and used is a vital component of effective policy and program decision making. By reducing barriers in access, acknowledging the limitations of data collection practices, and prioritising collaboration, we can ensure that everyone has access to the best possible evidence, to make the best possible decisions.

Dr Megan Weier is a Research Fellow at the Centre for Social Impact, and the lead researcher on the Australian Social Progress Index. You can follow Megan on Twitter at @MeganWeier. Isabella Saunders is a Research Assistant at the UNSW Centre for Social Impact. She is a co-researcher on the first Australian Social Progress Index. You can follow Isabella on Twitter at @isabellasaund