11 Data source and enrichment methodology

11.1 Data source and scope

The source data for this report covers every offender with any Corrections management history since 1980 and includes all of those offenders' careers, even the parts of those careers that occurred before 1980. No information in this report is based on surveys or estimates. It is based on historical operational records as stored electronically in Corrections' databases (and supplemented with some historical ethnicity data from the Ministry of Justice).

The main source of data has been the Offender-Major-Management-Period tables stored in the CARS (Corrections Analysis and Reporting System) data warehouse. These CARS tables in turn are derived from data stored in Corrections IOMS (Integrated Offender Management System) database, which is Corrections' day-to-day operational database. IOMS has been in use since mid 1998 for prisons and early 1999 for community probation services. At the start up of IOMS, historical data was imported from Corrections' legacy databases and also from the Law Enforcement System (LES). LES was originally known as the Wanganui Computer system. It was a Justice sector-wide system that was used nationally from 1976 until after 2000 and it was eventually decommissioned in 2005. When LES started in 1976 all new offender, offence and sentence details were entered into the system and a back-loading exercise took place to ensure current offenders at the time were entered along with all their previous history. The author understands that for some years after the initial start-up, as recidivists came to the attention of the sector, their new offences were loaded and also an effort made to enter the offenders' earlier offence and sentence history.

11.2 Data enrichment and simplification

The new Offender-Major-Management-Period tables that are used as the basis of this report are the result of considerable enrichment and simplification of the underlying data available in the IOMS database.

The goal has been to provide a single unambiguous timeline for each offender's career, which describes the sequence of major management states to which the offender has been directed. The new data-set enforces a one-day/one-status requirement for each offender. This is a huge simplification compared to the complexity of the data and overlapping directives in the lives of some of the offenders. However it provides the basis for a useful big picture analysis in which the numbers are self consistent and so that data issues can be identified and fixed.

To achieve the one-day/one-status requirement, a trumping process (see the major management categories in rank order) has been introduced that provides the "major management category" in situations where the data indicates several things are happening simultaneously. It must be recognised that this means that exact alignment with many other Corrections' reports is difficult. For example, under the trumping process a supervision sentence takes precedence over a contemporaneous community work sentence and a remand warrant takes precedence over a community sentence. In the timeline created the unmanaged periods between sentences and orders are also available for analysis, allowing the introduction of the concept of the "recently managed offenders' pool".

The diagram above represents all the different sentence and order directives recorded for a single offender. This representation demonstrates how these records can overlap in time and shows the hierarchy (on the vertical axis) applied to these records to produce a simplified non overlapping timeline. The "simplified" timeline can be read in conjunction with the key supplied, with colours indicating the nature of the offending and the height of the coloured blocks indicating the cost of the period of management. Thus the offender shown above was managed with a period of supervision for drug offending, followed by remand for burglary with a resulting community detention sentence, but then reverting to community work still to be completed for the drug offending, next the offender was classified with a status of "recently managed offender pool", but was later remanded for receiving with a subsequent prison sentence. However while still in prison a long term sentence for assault was added, from which the offender was eventually released first to home detention, then parole.