What works: a new model of public service delivery
Policy Adviser, Department of Corrections
Joshua Thurston is a Policy Adviser in the Strategic Policy Team. He joined the Department of Corrections in 2014 as a Probation Officer. He studied Philosophy at the University of Auckland.
“We often have no idea whether the things we do in government actually work or not, and achieve their stated goals. This is a disaster.” – Ben Goldacre, author of Bad Science (Goldacre, 2012).
In a climate of austerity, governments have become less willing to spend money where outcomes are uncertain or untested. Though medical treatment, crime reduction, and education are all worthy aims, the vehicles for achieving those aims have not always delivered as hoped.
With this greater push for results, a new model for determining which interventions to support has become the flavour du jour. Evidence-based policy is the new mantra which public servants and politicians across the world are following. New institutes, think-tanks, and lobby groups have been established to spread their preferred policies backed up by their evidence – but some believe that evidence-based policy is little more than a catch-cry, preventing honest engagement with the ideas presented.
The United Kingdom is world-leading in its commitment to the use and propagation of evidence-based policy. In 2010 the Cabinet Office established a Behavioural Insights Unit, known as the “Nudge Unit” after the 2008 book of the same name (Thaler & Sunstein, 2008). This team uses the disciplines of economics, psychology and public policy to test, trial and develop new or redesigned public services for better outcomes.
The United Kingdom has also established a network of seven What Works Centres – across fields as diverse as health, education, economic growth, and crime reduction. These centres have been set up as independent advisory bodies in their fields. The establishment of the What Works Network (the network) is grounded in the use of evidence-based policy, fiscal responsibility, and local decision-making (HM Government Cabinet Office, 2013). Each of the What Works Centres will focus on six key tasks within its area of policy:
- generating evidence synthesis (based on existing research)
- producing and applying a universal method for comparing the effectiveness of interventions
- putting the needs and interests of research users at the centre of its work
- publishing and disseminating findings in an understandable and usable format
- identifying research and capability gaps and working to fill them
- advising those undertaking research and projects to ensure their work can be evaluated effectively.
The use of evidence-based policy is not itself new – however, the level of commitment to it, and its pairing with independent advisory bodies, local decision-making, and operational focus make it a formidable development in the policy world of the United Kingdom.
Nothing works: rehabilitation or incapacitation?
In 1974, Robert Martinson authored a paper titled What Works? – Question and Answers about Prison Reform (Martinson, 1974). This paper concluded that “the present array of correctional treatments has no appreciable effect – positive or negative – on the rates of recidivism of convicted offenders”. Cullen and Gendreau note that this review “gave legitimacy to the anti-treatment sentiments of the day; it ostensibly ‘proved’ what everyone ‘already knew’. Rehabilitation did not work” (Cullen and Gendreau, 2000). Spencer claims that the political response was to move resources from rehabilitation to incarceration, but that researchers were equally quick to move into more sophisticated analysis to develop better information about what does work in rehabilitation (Spencer, 2013).
The United Kingdom and the United States for a period firmly espoused this “nothing works” view of rehabilitation, and the governments of Thatcher and Reagan held hard-line views on crime (Hollin, 2000). Martinson himself later went on to recant the conclusions of the 1974 paper. In the mid 2010s, crime and rehabilitation are again at the forefront of public policy debates, with David Cameron expressing a need for better provision of rehabilitation in prisons (Cameron, 2016). The debate has come full circle, and the United Kingdom is now embracing the use of evidence to support crime reduction.
The What Works Centre for Crime Reduction
The What Works Centre for Crime Reduction (the Centre) is based inside the College of Policing (the College). The College is focused on activities across three key areas of policing; knowledge, standards, and education. The Centre complements the existing functions of the College, while enhancing its ability to collate, distil, and propagate research and analysis of effective crime reduction policies and practices. A crucial measure of success for the Centre and the College itself is how the research and information they develop and provide is then incorporated, accepted, and actioned within police forces across England and Wales.
Evidence-based policy is of little value if poorly implemented or understood by decision-makers. The Centre attempts to address this potential limitation through its key outputs, a Crime Reduction Toolkit, and a series of What Works Briefings. These are targeted towards operational decision-makers, providing practical, accessible and expert advice on what exactly has been found to be effective in reducing crime – one topic at a time.
The Crime Reduction Toolkit
To create the toolkit, the Centre uses meta-analysis to examine research on particular interventions, incorporating both primary analysis and literature reviews, to create a synthesis of what all available evidence tells us about particular methods of reducing crime. In this way, the Centre produces relevant and up-to-date information about interventions used across the world, their effectiveness, and the necessary details about implementation. An important element of the toolkit is that it examines where and how interventions work, recognising that success is not simply a matter of replicating previous examples, but adapting them to one’s own context. A common concern about evidence for interventions is about reproducibility and validity in a new context. The Centre attempts to address this through evaluating what information studies provide about implementation.
The toolkit currently consists of reviews and evaluations of 35 different types of intervention, and their effectiveness in reducing crime. New reviews are added regularly. Interventions assessed to date include restorative justice conferencing, electronic monitoring, alcohol ignition interlocks, and street lighting. The toolkit is not simply a list of all available intervention with a tickbox – though it is that. Each intervention is assessed against five criteria for an overall evaluation. The scoring looks not only at whether the intervention has been shown to reduce crime, but also the quality of the data available, and how useful it will be to decision-makers. These criteria are:
- impact on crime (effectiveness)
- how it works (theory)
- where it works (context)
- how to do it (implementation)
- the economic argument (cost / benefit analysis).
These criteria can provide a guide to the robustness of the evidence for an intervention, and highlight potential gaps in evidence. Not only does the Centre provide guidance as to which interventions are effective, it also identifies those that research shows to be ineffective or, worse, actually increase offending. “Scared straight” style programmes and youth-involvement in the adult justice system have both been identified as potentially increasing criminal activity in participants – a perverse outcome which demonstrates the value of research in this area. Without robust evaluations of programmes, ineffective or harmful interventions may be propagated.
The Centre also provides a “research map” where it tracks research across the United Kingdom about crime reduction, providing a centralised location for academics and researchers to make contact and access relevant projects.
The Centre will be evaluated over a three-year cycle. The first evaluation was published in February 2015, as a baseline of knowledge about the College and Centre, and of the use of research in policing. The findings were promising; most interviewees were regular users of research, and believed that the Centre held promise for the future of policing. The evaluation also found that to be effective in propagating research and increasing the use of evidence by practitioners, the Centre would need to be a long-term resource, building a reputation and culture of quality outputs. In this account, decision-makers are reluctant to change practice or develop new methods of policing if they view the source of evidence or guidance as unreliable, “fad-like”, or risky. Therefore, the Centre is in a position of not only producing and collating evidence and research, but also of needing to engage with the policing community and Government to ensure that their mandate and role is understood and supported at all levels (Hunter, Wigzell, May & McSweeney, 2015).
Nudging: randomised controlled trials
While the Centre for Crime Reduction largely limits itself to collation and analysis of existing research, along with academic partnerships, the Behavioural Insights Team is deeply involved in actively identifying new ideas and innovative policies – and trialling them (Haynes, Service, Goldacre & Torgerson, 2012). They have undertaken pilot schemes trialling various models and implementations to identify which is effective at producing the desired outcomes (Behavioural Insights Team, n.d.).
The core of the theory behind the Behavioural Insights Team is that regulation and policy decisions can be unresponsive to normal human behaviours and beliefs, and that smarter regulation or better designed services can have dramatically different outcomes. In pursuit of better outcomes which preserve choice and freedom for citizens, they undertake trials and experiments – with robust checks and balances – to determine how public services can be delivered effectively.
Experiments have taken place in areas as varied as organ donations, hospital appointments, tax compliance, and charitable giving. With scaled, scientific trials, this team can say with authority that they have identified what models will result in the right outcomes. Not only are they testing whether current models are effective, they are comparing them with alternative methods to ensure opportunities and gaps are identified.
The West Midlands Police Force in England has partnered with the Behavioural Insights Unit. Chief Superintendent Alex Murray notes “Embedded change will only take place, however, when the hearts and minds of police leaders embrace evidence-based approaches themselves.” Murray is also heavily involved with the Society of Evidence-Based Policing, a charity which seeks to ensure research evidence is communicated, produced, and used – much like the What Works Centre aims to achieve (Ruda, 2015).
Limitations of the model
Where is the evidence?
Whyte takes issue with the notion of evidence-based policy altogether. He contends that interest groups promote studies which align with their pre-existing views, that many scientific studies are far from rigorous and ignore costs and changes in behaviour, and that many experts are held up as having answers to complex issues when their actual field of expertise is much narrower (Whyte, 2013). He provides the example of climate scientists whose views as to which policies to implement are treated with more respect than warranted, given that the scientist is not an expert in political matters, nor in economics, nor international affairs.
Whyte is further concerned by his view that political discourse is stifled by the use of evidence-based policy. Opponents of the evidence or those who question models used are portrayed as anti-science, and he argues that there are so many instances where evidence-based policies have had serious flaws that the term is no more than a panacea to provide a defence for expensive and interventionist policies. Whyte is correct in stating that appealing to evidence of any sort is not sufficient for a policy to be implemented; the evidence must be reproducible, rigorous, and relevant. However, examples where this has not been the case are not reason to claim that evidence-based policy is implausible as a model.
That doesn’t work where I’m from
Tseng, in The Uses of Research in Policy and Practice, identifies some potential barriers to the use of research and evidence in decision-making (Tseng, 2012). A range of legitimate concerns with the relevance of research are expressed, as well as some differing ideas about what is defined as evidence. Practitioners tend to place value on research that bears direct relevance to their location, demographic, or other variables, while being sceptical about the possibilities of cherry-picking results and studies to prove conclusions.
Decisions must still be made
In a technocratic model of government, public fears about crime or healthcare would not influence decision-making. Evidence would determine the appropriate course of action; which medicines to fund, which offences to target and how to improve educational outcomes. However, in the United Kingdom, as in New Zealand, politicians make the ultimate calls about what acts are crimes, what priorities Police will address, and how medical care should be distributed among the population (Rutter, 2012). By one measure, this is a limitation for evidence-based policy; public pressure and political necessities may scuttle potentially promising policies (O’Malley, 2013). By the same token, some measures may be effective and recommended, but have costs in terms of freedom or choice that are too important to overrule.
What next for evidence-based policy?
Unlike the fields of medicine or engineering, where new techniques for treatment or development are only advanced when proven safe and effective, crime reduction and policing are heavily politicised and use of solid evidence and research in decision-making is not as embedded.
Context matters. Practitioners raise legitimate concerns about the validity of findings from different contexts. Concerns have arisen about the reproducibility of psychological studies. People respond in ways that can be unpredictable. The Centre does not claim to have found a silver bullet for preventing crime. What it offers is a roadmap: offering practitioners evidence of what has been studied to date, providing academics opportunities for grants and projects, and demanding rigour in analysis in a sector where this is a recent development.
We await with interest the results of the Year Two evaluation of the Centre. Reliable evidence for crime reduction techniques in policing is one part of the test; implementation will remain a challenge.
Behavioural Insights Team. (n.d.). Retrieved April 06, 2016, from http://www.behaviouralinsights.co.uk/ - website of the Behavioural Insights Team, formerly part of the HM Government Cabinet Office
Cameron, D. (2016, February). David Cameron’s Prison Reform Speech. Retrieved March, 2016, from http://www.politics.co.uk/comment-analysis/2016/02/08/cameron-prison-reform-speech-in-full
Cullen, F., & Gendreau, P. (2000). Assessing Correctional Rehabilitation: Policy, Practice, and Prospects. Policies, Processes, and Decisions of the Criminal Justice System, 3.
Goldacre, B. (2012, June). Here’s our Cabinet Office paper on randomised trials of government policies. Read it. – Bad Science. Retrieved March, 2016, from http://www.badscience.net/2012/06/heres-a-cabinet-office-paper-i-co-authored-about-randomised-trials-of-government-policies/
Haynes, L., Service, O., Goldacre, B., & Torgerson, D. (2012, June 14). Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials. Retrieved March, 2016, from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/62529/TLA-1906126.pdf
HM Government. What works: evidence centres for social policy. London: Cabinet Office; 2013. Available from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/136227/What_Works_publication.pdf
Hollin, C.R. (2000). To treat of not to treat? A historical perspective. In C.R. Hollin (Ed.), Handbook of offender assessment and treatment (pp 3-15), Chichester, U.K.: Wiley.
Hunter, G., Wigzell, A., May, T., & McSweeney, T. (2015, February). An Evaluation of the ‘What Works Centre for Crime Reduction’ Year 1: Baseline. Retrieved March, 2016, from http://whatworks.college.police.uk/About/Pages/default.aspx
Martinson, R. (1974). What Works? - Questions and Answers About Prison Reform, The Public Interest, 35: 22-54
O’Malley, J. (2013, October). Damian McBride’s war on evidence-based policy. Retrieved March, 2016, from http://www.newstatesman.com/uk-politics/2013/10/damian-mcbrides-war-evidence-based-policy
Rutter, J. (2012, September). Evidence and Evaluation in Policy Making: A problem of supply or demand? Retrieved March, 2016, from http://www.instituteforgovernment.org.uk/sites/default/files/publications/evidence and evaluation in template_final_0.pdf
Institute for Government
Ruda, S (2015), Innovation in Policing: A guest blog from Chief Superintendent Alex Murray. Retrieved March, 2016, from http://www.behaviouralinsights.co.uk/home-affairs-and-security/innovation-in-policing-a-guest-blog-from-chief-superintendent-alex-murray/
Spencer, L. (2013). Evidence Based Practices Work. Corrections Today, 75(4). Retrieved from http://www.aca.org/aca_prod_imis/docs/Corrections Today/Commentary_Sept2013.pdf
Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT: Yale University Press.
Tseng, V. (2012). The Uses of Research in Policy and Practice. Sharing Child and Youth Development Knowledge,26(2). Retrieved March, 2015, from http://files.eric.ed.gov/fulltext/ED536954.pdf
Whyte, J. (2013). Quack Policy: Abusing Science in the Cause of Paternalism. SSRN Electronic Journal SSRN Journal. doi:10.2139/ssrn.2318507