What works in correctional rehabilitation? Lessons from 15 years of programme outcomes analysis
Dr Peter Johnston
Director Analysis and Research, Department of Corrections
Dr Peter Johnston has been with the department for over 20 years. He started in the Psychological Service in Christchurch, as one of three psychologists who set up the first Special Treatment Unit, Kia Marama, at Rolleston Prison in 1989. He then moved to PrisonServices, where he was involved in setting up a prison-based “inmate assessment centre”. In his current role he leads a team ofseven staff who undertake research and evaluation, and in-depth analysis of data, to support new policy initiatives.
Offender rehabilitation has always been central to the objectives of the Department of Corrections since its inception in 1995. We have made a wide range of rehabilitative services and interventions available to offenders, and implemented comprehensive frameworks of assessment, eligibility and referral to ensure that “the right offenders are matched with the right programme at the right time”.
It has always been understood that effectiveness in rehabilitation – that is, achieving tangible reductions in re-offending as a result of participation in interventions – can never simply be assumed. In reality, a number of processes and influences can limit or neutralise the beneficial effects of correctional rehabilitation. Examples include inappropriate targeting of offenders, inadequate facilitator skill, and adverse events and processes within the wider custodial environment.
To ensure that the investment in rehabilitation is used to best effect, outcome evaluation has been a priority for the department since the early 2000s. The statistical methodology for analysing programme impacts – the Rehabilitation Quotient (RQ) – was developed in 2001, and has been deployed annually ever since, with the results published in the department’s annual reports*.
One of the strengths of the RQ approach is that it can distil the specific contribution that an individual programme type makes to overall reductions in re-offending. It seeks to answer the question: “To what extent did this recent instance of correctional rehabilitation, with this particular cohort of offenders, have the desired effect in reducing re-offending?”
This article sets out some of the more important lessons that have been drawn from the results of these annual outcomes analysis exercises over the last 15 years. As will be discussed towards the end of this article, having evidence to answer questions about programme effectiveness has great value.
The range of interventions able to be “RQ’ed” each year has grown significantly since the early 2000s, when just a handful of core programmes were analysed. Currently, around 40 distinct rehabilitation/ reintegration services are included in the annual RQ process, with reconviction and reimprisonment “effect sizes” reported for each intervention. The annual expenditure on rehabilitation has increased commensurately. Being able to demonstrate that public funds are being expended to good effect is crucial.
Rehabilitation Quotient analysis is, of course, just one of the ways in which the department obtains information on programme effectiveness. Also important in this area are fieldwork-based evaluations and reviews, which gather detailed evidence concerning quality of delivery and participant response.
Lesson 1: Overall, most of what we are doing to reduce re-offending, succeeds. Under routine delivery across the general offender population, our rehabilitative interventions have modest but positive impacts on re-offending. Effect sizes (ESs) are mostly in the 3 – 8 percentage-points range. For a programme with the latter ES this means that, instead of an “expected” rate of reimprisonment after 12 months of 25 percent, a programme’s participants would have an actual reimprisonment rate of 17 percent. ESs of this magnitude are not huge – correctional systems the world over seldom get dramatic reductions in re- offending as a result of their programmes – but they are meaningful and significant reductions. And generally they mean that the benefits of programmes (in terms of reduced costs of future offending) are greater than the costs of delivering the programme.
This finding is particularly important for two reasons. Firstly, the research literature mainly consists of studies focused on “flagship” programmes delivered under “ideal” conditions (well-resourced, highly trained facilitators, oversight by expert practitioners and academics). It is not particularly difficult to obtain good results when rehabilitation is delivered under optimal conditions; it is, however, considerably more challenging to get good results when delivery is “routinized” and broadly implemented across an entire national correctional system.
Second, the results we achieve are consistent with what is achieved in other countries. For example, a large-scale outcomes analysis of a specific cognitive- behavioural programme, delivered to over 20,000 prisoners in United Kingdom prisons between 1998 and 2005, found an overall ES of 8.4 percentage points (Sadlier, 2010).
Lesson 2: New programmes seem to require a “bedding-in” period before demonstrating impact, usually of two to three years. Low initial ESs have been observed on several occasions, such as with the new Medium Intensity Rehabilitation Programme forma in 2007-08, the introduction of the 3-month format of the Drug Treatment Unit (DTU3) programme in 2011, and with the Special Treatment Unit Rehabilitation Programme (STURP). All of these programmes are now considered successful, reliably reducing re-offending by between four and 13 percentage points.
This bedding-in effect is likely to be associated with staff developing their skills, gaining confidence in their roles, learning to work together as a team, becoming more familiar with programme materials, and so on.
Lesson 3: There tends to be a direct and positive relationship between intensity of programme (i.e., number of hours of face-to-face facilitator-participant contact – sometimes called “dosage”) and magnitude of ESs. A good example of this is the DTUs, where the six month format almost invariably produces ESs that are larger than those of the three month format. Also, as noted below, we tend to obtain our greatest impacts from our most intensive programmes – particularly the STURP. The STURP is over nine months in duration, and involves hundreds of hours of group work and face-to-face engagement with psychologists and programme facilitators.
On the other hand, surprisingly good results are often obtained from some brief interventions, such as the community Short Rehabilitation Programme (SRP – 24 sessions over six-eight weeks); even the Short Motivational Programme (SMP – five sessions over up to five weeks) makes measurable impacts on the reconviction rates of prisoners and community offenders.
Lesson 4: Correctional rehabilitation has potential to make participants worse off. A programme known as “Straight Thinking” was delivered between 1997 and 2005, as an NZ adaptation of a “cognitive skills” course developed overseas. Participants in Straight Thinking were found to be reconvicted and re-imprisoned at rates several percentage points higher than comparison offenders. As can be imagined, these results were met with considerable scepticism at first, and it was assumed that the RQ method itself was faulty. However, the same results were reproduced after the RQ method had been revised and refined (2004), following which the programme was discontinued.
Investigations indicated that this adverse outcome resulted from a combination of factors: the lack of offence focus in the programme’s content, inadequately trained and supervised facilitators, and a programme group environment that was not conducive to personal change. These findings led to significant changes to the entire rehabilitation suite of programmes, including ensuring that programmes now have an appropriate offence focus, facilitators are better trained, and group environments are supervised and monitored to ensure they remain conducive to positive personal change.
Lesson 5: A similar lesson learned at that time was that adequately designed but poorly-delivered programmes often fail to generate measurable impacts. The pre-MIRP “100-hour programme” (of which there were four variants – adaptations for violence, alcohol and other drugs, driving and general) was adequately designed, but delivery was often sub- standard, resulting in several years of near-zero ESs. This programme was subsequently overhauled and re-designed, and re-launched as the MIRP.
Other innovations that seem to have merit can also fail. An example is the “Faith-based Unit” at Rimutaka Prison, which lacked the critical ingredients necessary for a positive impact on re-offending, and was closed as a result.
These findings underline the reality that multiple things need to “go right” for positive impacts to occur: good programme design, skilled and motivated facilitators, sound selection of participants, a stable/supportive environment within which delivery occurs, and good retention rates of participants. It is, of course, an on-going challenge for all correctional agencies to get the best out of their rehabilitation investment, but continuously working on improving programme “integrity” – i.e., the programme delivered to offenders conforms closely to the design and approach which is intended – is critical.
Lesson 6: Educational courses and employment/ industry training now reliably produce positive impacts on re-offending (ESs usually between 3 – 5 percentage points, and statistically significant). Reintegrative support services are also achieving good results, especially the “Out of Gate” (OOG) service. Further, we have seen an interactive effect, whereby prison programmes followed by OOG lift the measured impacts above those achieved by either programme or OOG alone. This may come as a surprise to some who have tended to assume that only psychological-style programmes are effective, but we now have many years of evidence to confirm that educational courses and employment/industry training can reliably produce statistically significant positive impacts on re-offending.
Lesson 7: In recent years we have begun to run RQ analyses on specific programmes with differing participant types. This can only be done where the cohort numbers are sufficiently large, but we have done this with a number of mainstream rehabilitation programmes. Analysis of the participant data shows that Māori participate in our mainstream rehabilitation programmes in numbers equal to the proportion of Māori in the offender population. Secondly, Māori complete mainstream prison programmes at rates equal to non-Māori, and at a slightly lower rate in the community. The RQ analysis suggests that impacts for Māori are equal to, and sometimes better than, those recorded for non-Māori.
Further, we have shown that gang members (70% of whom are Mäori) also participate in programmes in reasonably significant numbers, and are benefitting from doing so. Although re-offending rates remain high, gang members who participate are likely to re-offend less seriously, or to go for longer without re-offending.
Lesson 8: Specifically culturally-based interventions – Maori Focus Units, the Pacific Focus Unit, tikanga courses, and the “Bi-cultural Therapy Model (BTM) – when delivered as “standalone” interventions, tend to produce only small ESs. Awareness of this has led to the redesign of the Māori Focus Units model into Te Tirohanga which will bring these units more into line with established principles of effective correctional rehabilitation. These forms of rehabilitation also have motivational value, in beginning to orient offenders to the tasks and challenges of personal change.
Lesson 9: The RQ method, using a 12 month follow-up period, and counts of general re-offending, is not well- suited to evaluating child-sex offender programmes. This is because sexual re-offending, if any occurs, will generally not show up in official records for several or many years after release. We nevertheless include the two child sex offender special treatment units (Te Piriti and Kia Marama) in the annual RQ round, and tend to find small but consistent reductions in general re-offending. This provides some assurance of on-going programme soundness.
Lesson 10: Finally, the standout lesson from the experience of outcomes analysis is that, when done properly, correctional rehabilitation can be very successful. The case in point is the STURP, which is our best-performing rehabilitation programme. This programme regularly produces reductions in reimprisonment rates of 12 percentage points and more, and reductions in reconvictions (which include convictions resulting in either prison or a community sentence) of up to 17 percentage points.
The STURP results provide further confirmation that close adherence to known principles of effective correctional rehabilitation can generate excellent results. This is particularly impressive given the target group for the programme; high-risk violent offenders, many of whom are gang-affiliated, and most of whom are challenging to work with. In other words, the programme succeeds with our toughest customers. Delivering these kinds of results year on year really is an outstanding achievement.
RQ results are used for a range of purposes in the overall management of the department’s business. Critically, they ensure that we can have confidence that our programmes are effective. On the flip side, ineffective services (such as Straight Thinking) can be identified and discontinued, with funding re- directed into more effective programmes. Results have also served to identify programmes that require strengthening, as revealed by weak or variable ESs. They have enabled us to improve matching of offenders to programme type, for example, the short (three month) and longer (six month) formats of the Drug Treatment Units. RQ also permits formal cost-benefit analysis, where costs of programme delivery are weighed against “future costs avoided” through reduced crime and victimisation. This in turn can be used to generate the kind of hard evidence now required to support funding bids for rehabilitation expansion.
New Zealand remains the only country in the world that routinely measures and reports on the outcomes of the full suite of its rehabilitative interventions. The process has major benefits in enabling us to direct, and re-direct, resources to where we get best effects, to improve effectiveness, and to avoid wasted effort.
* For a detailed description of the RQ methodology, see an earlier article by the author in “Practice” here: http://www.corrections. govt.nz/resources/newsletters_and_brochures/journal/ volume_1_issue_1_may_2013.html
Sadlier, G. (2010). Evaluation of the Impact of the HMPrison Service Enhanced Thinking Skills Programme on Reoffending: Outcomes of the Surveying Prisoner Crime Reduction (SPCR) Sample (Ministry of Justice Research Series 19/10). London: Ministry of Justice.