Risky business: Evaluating the Dynamic Risk Assessment for Offender Re-entry for use with New Zealand youth
Jonathan Muirhead, Clinical Psychology Student
Chief Psychologist’s Team, Service Development, Department of Corrections
Dr Clare-Ann Fortune, Senior Lecturer in Clinical Forensic Psychology
Victoria University of Wellington
Professor Devon Polaschek, Professor of Criminal Justice Psychology
Victoria University of Wellington.
Jonathan (Jono) Muirhead has been with Corrections for two years. He started in the Chief Probation Officer’s team, where his role mainly focused on conducting research and helping write training material. Jono was offered a scholarship by the Department to complete his clinical psychology studies in 2018, and he is now working part-time in the Chief Psychologist’s team while he studies.
Clare-Ann Fortune, PhD, PGDipClinPsyc, is a senior lecturer in clinical forensic psychology at the School of Psychology, Victoria University of Wellington. She is a registered clinical psychologist and teaches on the Forensic Psychology and Clinical Psychology programmes. Her research interests focus on risk assessment and rehabilitation for youth involved in the justice system, and factors impacting the participation of young people in the youth justice system.
Devon Polaschek, PhD, DipClinPsyc, is a professor in the School of Psychology, and the Interim Joint Director of the New Zealand Institute of Security and Crime Science at the University of Waikato. Her research interests include understanding and preventing re-offending in serious violent and sexual offenders, family violence, psychopathy, imprisonment, desistance, reintegration and parole.
Key words: Youth, DRAOR, Offending, Probation, Risk Assessment
The first author acknowledges the supervisors of his master’s thesis, Dr Clare-Ann Fortune and Professor Devon Polaschek at Victoria University of Wellington, for all of the advice they gave in helping complete the thesis this article was based on. Thank you as well to Dr Nick Wilson and Nikki Reynolds for the assistance in editing my thesis from 109 pages into something much more appropriate for a practice journal.
This paper presents a selection of the findings of this thesis research project, which the first author completed at Victoria University of Wellington, in fulfilment of the requirements for the degree of Master of Science in Forensic Psychology. For the full thesis report, see http://researcharchive.vuw.ac.nz/handle/10063/5221
Imagine you are a probation officer given the job of assessing an individual named Ben for his risk of reconviction while he serves a community sentence. Although Ben continues to consort with the friends who originally got him into trouble and is still struggling with anger and impulsivity issues, he has stopped abusing drugs and alcohol, and is very responsive to your advice as his probation officer. How likely do you think Ben is to be reconvicted? Does his age matter? And what if Ben’s attitude or situation changes? Would updated information improve your evaluation of the risk he poses?
Risk assessment is an important area of forensic psychology, and a lot of work has been done identifying factors that influence someone’s likelihood of re-offending. The majority of this work in correctional settings has focused on adult males, with the assumption that these findings will translate to other populations of people who offend (e.g. youth and women; Singh, Grann, & Fazel, 2011). However, despite a large overlap of factors relating to offending for different groups, there are also a number of differences that are often overlooked (e.g. mental health is a stronger risk factor for youth offending, compared to adult offending where it is only weakly predictive; Borum, 2003). Some factors are more influential at different points in one’s life (e.g. peers have been found to be a stronger risk factor during adolescence than adulthood; Hoge, Vincent, & Guy, 2012) which could have an impact on a risk assessment’s accuracy.
Despite a number of risk assessment tools having been developed in recognition of the differences between populations, and despite studies having validated many risk assessment tools for different populations, there is still more work to be done. One population suffering from a lack of research is older youth (17-19 years old), leaving uncertainty as to whether they should be assessed as children or adults and which measures should be applied.
Currently in New Zealand (NZ), the Dynamic Risk Assessment for Offender Re-entry (DRAOR; Serin, 2007) is being used by the Department of Corrections to assess people being managed in a number of situations. This article is based on a master’s thesis (Muirhead, 2016) that looked into the DRAOR’s use with youth who were aged 17-19 and were serving a community sentence of between 6 and 18 months. Community sentences are of particular importance when considering those under 20 years old, as this is an age when many people commit a large number of offences (Moffitt, 1993; Loeber, Farrington, Stouthamer-Loeber & White, 2008) and if caught, may end up on community sentences. Almost 5,000 youth started community sentences in 2013 (Department of Corrections, 2013).
Understanding what the DRAOR can tell us about risk, risk prediction, and changes in risk is valuable across a range of NZ correctional populations. In addition to the value of examining the DRAOR generally for youth, there is current interest in whether more recent assessments of risk are more predictive than older ones. Aspects of the study summarised here address two components of the research: 1) How well do initial DRAOR assessments predict reconvictions in a youth community sentence sample? and 2) Do more up-to-date DRAOR scores out-perform initial scores in predicting reconviction?
An archival dataset was provided by the New Zealand Department of Corrections for use in this research. The dataset contained anonymised information on a sample of male and female youth (<20 years old) who served a community supervision sentence of 6-18 months between 1 January 2011 and 31 December 2013. The initial dataset provided by Corrections contained information about 547 youth who had been assessed with the DRAOR during their community sentence. After some exclusions (e.g. due to incomplete data, outside the age range), the final sample for analysis had 398 youth. The sample was predominately male (81.9%; female = 18.1%), with the highest ethnic representation being for Māori (48.2%; European = 37.4%, Pasifika = 8.8%, Asian = 0.8%, and Other = 4.8%).
Dynamic Risk Assessment for Offender Re-entry (DRAOR).
The DRAOR (Serin, 2007) is a risk assessment tool designed for use with people serving community sentences or parole (Serin, 2015). The DRAOR has been fully implemented in NZ since April 2010 and research is being done with the DRAOR in Australia, NZ, Canada, and a few US states.
Probation officers regularly meet with people who have been convicted and are living in the community on sentence. These officers complete a DRAOR assessment in each meeting. The assessments are conducted by way of an interview with the person, as well as taking third-party information, such as police records or information from family members, into account. These regular assessments are intended to allow probation officers to monitor a person’s risk of re-offending over time, to not only ascertain if the person is likely to re-offend, but also when (Serin, 2015).
The DRAOR contains 19 theoretically-derived dynamic risk and protective factors that are distributed across three subscales: Stable Dynamic Risk, Acute Dynamic Risk, and Protective (see Table 1 below). Each item is scored on a 3-point scale from 0 to 2. Although in practice these subscales are generally used to guide professional judgement of an individual’s level of risk, for research purposes the scores are often combined into a total score. The total score is the sum of the acute and stable risk scores minus the protective score. This allows for the total score to fall between a minimum of -12 (scoring 0 for each risk factors and subtracting 2 for each protective factor) and a maximum of 26 (scoring 2 for all risk factors and 0 for all protective factors).
The DRAOR is still relatively new and less researched than many risk prediction tools (e.g. Level of Service/Case Management Inventory). However, there have been a number of studies that have found the DRAOR to be reliable and to have predictive validity for a number of NZ populations including women, youth, and adults (AUC range: .62 - .74; Ferguson, 2015; Hanby, 2013; Lloyd, 2015; Scanlan, 2015; Tamatea & Wilson, 2009; Yesberg & Polaschek, 2014).
DRAOR subscale items
Responsive to advice
Attitudes towards authority
Opportunity/access to victims
Sense of entitlement
Attachment with others
How well do initial DRAOR scores predict reconviction in a youth sample?
DRAOR administrations begin very early on in an individual’s sentence. It is important to know how well these initial assessments perform when it comes to predicting criminal conduct much later in the sentence, or after it has ended. It was hypothesised that those who receive higher initial DRAOR risk scores and lower protective scores should be more likely to be reconvicted for a new offence, while those with lower risk scores and higher protective scores have a reduced likelihood of reconviction. In order to test how well the initial DRAOR scores predicted reconviction, univariate Cox regressions were performed separately on each of the three initial DRAOR subscale scores and on the total score.
Box 1: Predicting recidivism using Cox regression
We examined the predictive validity of the DRAOR subscales using Cox regression survival analysis: a type of analysis used when research participants don’t all have the same length of follow-up time. Each of three DRAOR subscales was used in its own analysis, as a variable that might predict recidivism. The dependent variable was recidivism and the time variable was the number of days to the first re-offence for those who were reconvicted within the study period, or the number of days until the end of the study period for non-recidivists.
Cox regressions provide us with a hazard ratio and an Area Under the Curve (AUC) output. The hazard ratio indicates the increased likelihood of a hazard (re-offending in this case) for every 1 point increase in the predictor variable (DRAOR subscale score in this case). For example, the hazard ratio of 1.09 for the Acute subscale in Table 2 indicates that for every 1 point increase in the Acute subscale, a person is 9% more likely to have been reconvicted.
The AUC for this study indicates the probability that a randomly selected member from one group (recidivists) will have a higher DRAOR risk score than a randomly selected member of another group (non-recidivists). An AUC of 0.50 would indicate the tool was no better than chance (50%) at distinguishing those who are reconvicted versus those who are not (i.e. it would be as accurate as flipping a coin). An AUC of 1.00 would indicate 100% accuracy. The generally accepted cut-off scores for the predictive ability of AUCs (Rice & Harris, 2005) are: low – 0.56 to 0.63; moderate – 0.64 to 0.70; and high – 0.71 and above. For this study, the AUC of .59 for the Acute subscale in Table 2 indicates that there is a 59% chance of a randomly selected recidivist having a higher Acute score than a randomly selected non-recidivist from this sample.
The results suggested that the initial DRAOR assessment is somewhat predictive of reconviction for youth serving community sentences. The AUC results (in Table 2 below) for these initial DRAOR scores were in the low-moderate range for predictive accuracy for any future reconviction (.63 or 63% accuracy for the total score; range of 57-69%). Another way to interpret an AUC is in terms of relative improvement over chance. Using this approach .63 means that the total DRAOR score provided a 26% improvement in decision making over chance (see Rice & Harris, 2005 for an in-depth discussion on the interpretation of AUCs).
Univariate Cox regression models for initial DRAOR scores predicting re-offending
Model for initial scores
Do more up-to-date DRAOR scores out-perform initial scores in predicting reconviction?
Serin (2007, 2015) recommends that the DRAOR be administered regularly to capture change in dynamic risk factors and protective assets. Theoretically, we expect that more recent, more up-to-date risk and protective scores should be better at predicting reconviction, since by definition they are made on information that is closer in time to the reconviction. In order to assess how well a more up-to-date DRAOR assessment predicts reconvictions, the most recent DRAOR subscales and total scores were analysed.
The predictive validity of the initial scores was compared to the more up-to-date proximal scores by entering the two scores sequentially into a series of four Cox regressions as before: one for each subscale, and one for the total DRAOR score. Table 3 shows that the proximal assessments were all significant predictors of recidivism, and in fact, once the proximal scores were used, the initial scores were no longer significant predictors. The largest hazard ratio from these models came from the proximal acute score; 1.22, indicates a 22% increase in offending for every 1-unit increase in proximal acute score. It should also be noted though that for the protective subscale, the overlapping CIs for initial and proximal scores, indicating that the proximal scores are not significantly better than the initial ones.
Multivariate Cox regression models for initial and proximal DRAOR scores predicting reconviction
1.00 [0.94, 1.06]
1.22 [1.15, 1.29]
0.98 [0.90, 1.06]
1.20 [1.13, 1.28]
0.96 [0.89, 1.04]
0.86 [0.81, 0.92]
0.99 [0.96, 1.02]
1.09 [1.07, 1.12]
This study found that initial DRAOR scores were somewhat predictive of future offending for youth assessed in the community with an overall low-moderate accuracy, but more up-to-date assessments were more predictive.
The original thesis also examined the DRAOR’s predictive ability for violent offending leading to reconviction. Although not recorded in this article, it was found that the DRAOR was not as good at predicting reconviction for a violent offence compared to any new conviction, but the proximal scores still fell in the moderate range of predictive accuracy. For a more in-depth look at these results please see Muirhead (2016).
Other facets of the full thesis that have not been reported here looked at 1) whether DRAOR scores for youth changed over time and, if they did, whether that change was predictive of reconvictions and 2) the predictive ability of the rate of change youths made on their DRAOR score. It was found that those who were ultimately reconvicted tended to have less improvement in risk and protective factors, compared to those who were not reconvicted. After finding that the amount of change was predictive of future convictions, we ran some more advanced statistical analyses that took the amount of time in the community into account. It was found that those who were not reconvicted improved at a faster rate per month while they were in the community than those who were reconvicted. Again, full details of these findings can be found in Muirhead (2016).
The results showed that the DRAOR is an effective risk assessment tool for probation officers to use with New Zealand youth serving community sentences. Not only did DRAOR scores predict future convictions, but it was found that the amount of change someone made over time was also indicative of future reconviction, regardless of the initial score.
That later DRAOR scores for youths on community sentences were more predictive of recidivism rates should encourage probation officers to continue to update DRAOR scores as they continue to work with youth on sentence. Up-to-date DRAORs help to indicate areas of concern, which probation officers can then target when working with youth to reduce their risk and build protective factors.
The research results support the use of the DRAOR with youth on community sentences in NZ. However, the DRAOR should continue to be validated, not just for youth, but for other populations as well. Since the DRAOR is used across New Zealand Corrections, as well as in a number of other countries, it is crucial that we increase our understanding of how the DRAOR performs. With a better understanding we can improve our confidence in its use, and also potentially refine the tool for different populations if it is found that particular items or subscales are better indicators of future behaviour for certain groups.
The assessment of youths’ risk of reconviction is an important area that is often overlooked, with most risk assessment research only looking at very young people or adults, neglecting those who fall in between. It is especially important to understand and monitor older youths’ levels of risk, given the high rates of crime for those in late adolescence (Moffitt, 1993). This research validated the DRAOR’s use with older youth (17-19 years) serving community sentences in NZ, which will allow for more confident use of the DRAOR with this population in the future.
In concluding, cast your mind back to Ben who was introduced at the beginning of this article. This research suggests that we can be confident in using the DRAOR to predict his risk of reconviction, even if he is only 17 years old, and we can also factor in the relevance of changes in scores in altering his reconviction risk. These findings are important due to the potentially serious consequences of an inaccurate risk assessment. Not only will we be able to use this information to guide the level of monitoring and intervention for Ben, but with more understanding of rates of change for the DRAOR we can look at intervening if, and when necessary to reduce the likelihood of further offences.
 Ethics approval was also granted by the School of Psychology Human Ethics Committee under the delegated authority of the Victoria University of Wellington Human Ethics Committee.
Borum, R. (2003). Managing at-risk juvenile offenders in the community: Putting evidence-based principles into practice. Journal of Contemporary Criminal Justice, 19, 114-137. doi: http://dx.doi.org/10.1177/1043986202239745
Department of Corrections. (2013). Trends in the offender population (2013 calendar year). Wellington, New Zealand: Retrieved from http://www.corrections.govt.nz/__data/assets/pdf_file/0012/738939/Trends_in_the_Offender_Population_2013_updated.pdf.
Ferguson, A. (2015). An age-old issue: Evaluating the applicability of adult criminal risk assessment tools for use with youth offenders. Unpublished master's thesis. Victoria University of Wellington, Wellington, NZ.
Hanby, L. (2013). A longitudinal study of dynamic risk, protective factors and criminal recidivism: Change over time and the impact of assessment timing. Unpublished doctoral thesis. Carleton University, Ottawa, Canada.
Hoge, R. D., Vincent, G. M., & Guy, L. S. (2012). Prediction and risk/needs assessment. In R. Loeber & D. P. Farrington (Eds.), From juvenile delinquency to adult crime: Criminal careers, justice policy, and prevention. New York, NY: Oxford University Press.
Lloyd, C. (2015). Can a dynamic risk instrument make short-term predictions in “real time”? Developing a framework for testing proximal assessment of offender recidivism risk during re-entry. Unpublished doctoral thesis. Carleton University, Ottawa, Canada.
Loeber, R., Farrington, D.P., Stouthamer-Loeber, M., & White, H.R. (2008). Introduction and key questions. In: R. Loeber, D. P. Farrington, M. Stouthamer-Loeber & H. R. White (Eds.), Violence and serious theft: Development and prediction from childhood to adulthood. New York, NY: Routledge.
Moffitt, T. E. (1993). Adolescence-limited and life-course-persistent antisocial behavior: A developmental taxonomy. Psychological Review, 100, 674-701. doi: http://dx.doi.org/10.1037/0033-295X.100.4.674
Muirhead, J. (2016). Risky business: Evaluating the Dynamic Risk Assessment for Offender Re-entry for use with youth. Unpublished master’s thesis. Victoria University of Wellington, Wellington, NZ. Available at http://researcharchive.vuw.ac.nz/handle/10063/5221
Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen's d, and r. Law and Human Behavior, 29, 615-620. doi: 10.1007/s10979-005-6832-7.
Scanlan, J. (2015). Assessing women’s risk of recidivism: An investigation into the predictive validity of the Dynamic Risk Assessment for Offender Re-entry (DRAOR) with matched samples of community-sentenced women and men. Unpublished master's thesis. Victoria University of Wellington, Wellington, NZ.
Serin, R. C. (2007). The dynamic risk assessment for offender re-entry (DRAOR). Unpublished user manual.
Serin, R. C. (2015). The dynamic risk assessment for offender re-entry (DRAOR). Unpublished user manual.
Singh, J. P., Grann, M., & Fazel, S. (2011). A comparative study of violence risk assessment tools: A systematic review and metaregression analysis of 68 studies involving 25,980 participants. Clinical Psychology Review, 31, 499-513. doi: 10.1111/j.1745-9133.2009.00551.x
Tamatea, A., & Wilson, N. (2009). Dynamic risk assessment for offender re-entry (DRAOR): A pilot study. New Zealand: Department of Corrections. Unpublished government report.
Yesberg, J. A., & Polaschek, D. L. L. (2014). Assessing dynamic risk and protective factors in the community: Examining the validity of the Dynamic Risk Assessment for Offender Re-entry. Psychology, Crime, & Law, 21, 88-99. doi: 10.1080/1068316X.2014.935775