Contracting for outcomes: simple KPIs not enough for complex problems

David Donaldson (@davidadonaldson) writes about new research that casts doubt on whether outcomes-based contracting solves a fundamental problem: unintended consequences. Listening to stakeholders, although no breakthrough, remains key. The systematic review of evidence on outcome-based contracting was conducted by Emma Tomkinson and published in ANZSOG's Evidence Base journal.

Three decades after outcomes-based contracting was introduced into US employment services, the jury’s still out on whether its use in human services really improves people’s lives.

This is despite outcomes-based contracts — a type of agreement where payment is made when certain targets are reached — often meeting the requirements specified in the agreement. So what’s the problem?

The failure of outcomes-based contracting to live up to the hype comes down to the fact that it’s quite difficult to reduce broad measures of wellbeing down to one or two key indicators, according to a recent evidence review by social impact analyst Emma Tomkinson for the Australia and New Zealand School of Government.

Misaligned targets

Outcomes-based contracting was introduced into American employment services in 1982, Australian employment services in 1998, and via several models in New Zealand in the late 1990s and early 2000s. Social impact bonds are a slightly tweaked version of this older concept.

Thanks to a lack of data, it’s unclear how outcomes-based contracting compares with other types of outsourcing, and with government-delivered services. Some studies have shown it tends to be cost-neutral, or cheaper, compared to other modes of delivery.

Nonetheless, there are some clear issues with the outcomes it delivers. The need to define targets clearly can lead to a black and white approach that belies the complexity of human services. Targets named in the contract, while being hit, often become separated from the original intention of the agreement’s framers.

The commonly resulting misalignment between targets and desirable holistic outcomes means that in many cases it’s unclear the end product of the contract is a genuine improvement in the client’s life. So while superficially the contract is meeting its ends, it may not be driving desirable outcomes for the end client.

And while specified outcomes are being met, studies show other measures of wellbeing commonly remain unchanged, and some even worsen. Because many of the social issues such contracts are set up to tackle — unemployment, child welfare, recidivism — are so complex, this can reflect a lack of overall progress.

For instance, a child welfare-related study found that when outcome payments related only to placing children in any family situation, placement rates rose overall, but the rate of placement with biological families — widely considered the optimal outcome — dropped.

In another case, an agreement with the UK’s Doncaster prison stipulated payment based on the proportion of offenders who committed one or more offences in the 12 months following release from prison. This incentivised the prison to keep in contact with prisoners after their release to help them improve their lives and avoid reoffending.

Unfortunately, because the prison would miss out on payment for every individual who had committed one offence, there was no incentive to continue to work with individuals after that point, and one study noted “some delivery staff reported frustration that support is withdrawn, undermining the interventions previously undertaken.”

In some cases a slight change can help deal with unintended consequences. A different prison agreement, based around the number of times an individual re-offended, had the more positive impact of incentivising the gaol to continue working with prisoners for the full 12 months post-release.

Perverse incentives squash flexibility

A common reason for implementing outcomes-based contracting is to increase flexibility and encourage individualisation in how providers deliver services, but the associated regulation of many agreements has perversely resulted in greater rigidity and standardisation.

The big challenge, Tomkinson thinks, is to figure out how to design contracts so payment-related indicators accurately reflect desired outcomes. Such agreements often start out simply, but as perverse or misaligned targets are corrected for, increasing complexity leads to unwieldy regulatory structures.

The impact of perverse incentives can be seen in the experience of Australia’s employment services system. Private companies are paid to help find unemployed people jobs, but different fee structures — which have been altered over the years thanks to exactly this problem — have created incentives for different unexpected outcomes.

Paying companies at the point when clients find a job can cause “cream skimming” — encouraging them to ignore more difficult clients in favour of those who don’t need much help finding a job. On the other hand, paying fees for every week someone is on the books — designed to increase investment in those who need to be skilled up with training and so on — can discourage providers finding jobs for clients.

“The balance between fees and outcome payments is hard to get right and employment contracts are frequently adjusted to correct for unintended consequences.”

So the system ends up using different variations in combination, and weighting payments at different rates, adding to the compliance required, as well as forcing providers to semi-regularly update their practices and understandings of the regime.

This red tape in turn leads to lower levels of flexibility and innovation in service delivery as staff worry about filling out forms incorrectly and getting in trouble with the department, studies have found.

Tomkinson also notes that although the main use of such contracts in Australia has been in employment services, there is no strong reason to believe it is particularly suited to this policy area, especially as the main driver of unemployment — economic conditions — has a far bigger impact on outcomes than contract design.

Time spent listening saves later

Some of the problems may reflect a lack of capacity in the public sector and/or service delivery organisations to work out effective agreements — a challenge Tomkinson thinks public sector managers should keep in mind.

The amount of resources put into developing these agreements is a problem. There appears to be “an inadequate investment of time and skill spent rigorously thinking through the intended and unintended consequences of new contracts”, she argues.

This may be due to the additional time requirements — or because the public sector doesn’t do a very good job of listening to stakeholders. In employment services, Tomkinson notes, “many of the problems arising were predicted by external organisations, but dismissed during the contracting process.”

Posted by @MsSophieRae.

Reference: Tomkinson, E. Outcome-based contracting for human services. Evidence Base, 2016(1): 1-20.