An integrated approach: Holistic assessment of vocational trainees
Education Programmes Manager, Department of Corrections
Graeme Couper has a degree in Agriculture from Massey University and spent 15 years as a dairy farmer. He has also been a tutor for agriculture and horticulture trainees, has owned and managed a primary-sector-based private training provider and, prior to coming to Corrections, was a moderator and then education manager at Primary ITO where he was responsible for managing the Targeted Review of Qualifications (TRoQ) for the primary sector. In November 2016 he completed a Master’s degree in Tertiary Education with Distinction. His thesis (Couper, 2016) was based on a research project on the new assessment approach taken with dairy farm trainees as the new qualifications were implemented into the agriculture sector. This article is based on the findings of that research project.
“Experience without theory is blind, but theory without experience is mere intellectual play” – Immanuel Kant.
In 2010, the New Zealand Qualifications Authority (NZQA) initiated the Targeted Review of Qualifications (TRoQ). The intended outcome of this review was to replace all national and local (provider) qualifications with a suite of New Zealand qualifications. The main drive for this review was to remove duplication of qualifications and to provide much more clarity among learners, their support networks and employers as to what the qualifications meant, how they related to specific job roles in industry and where they fitted into a learning continuum.
In January 2016, these new qualifications were introduced into the dairy farming sector by Primary ITO. In the spirit of the new qualifications taking a more integrated, graduate outcome focused approach, Primary ITO introduced a significantly different assessment methodology based on an integrated evidence portfolio.
In 2016, a research study into how effective this new integrated assessment approach had been was undertaken to contribute towards a thesis for a Masters in Education. This article outlines the findings of that research. It should be noted that the key findings of this research relate specifically to level 3 and 4 dairy farm trainees, however it is likely that the key principles discussed throughout this article will relate in general to most workplace-based vocational learning.
The conclusions from this research are very relevant to Corrections as at any given time there are over 2,000 prisoners engaged in industry training, across a range of prison-based industries. In most cases, these industries operate as typical commercial workplaces and the trainees are working towards national qualifications. The expectations of our trainees are the same as other industry trainees who are operating in real commercial workplaces. Any factors that this research has identified that will assist vocational learners to succeed are likely to apply equally well to prison-based learners.
Changes to qualifications
The key difference between the old national qualifications, and the new, New Zealand qualifications, is the focus on graduate outcomes rather than unit standards. The graduate outcome approach to the new qualifications focuses on the psychomotor, cognitive, and affective aspects of learners. During the TRoQ, and subsequent qualification development process, these graduate outcomes were represented as graduates’ skills, knowledge and attributes – or what a graduate should “know, do and be” having achieved the qualification. This shift was a significant departure from the national qualification system where learners were assessed on their competence in unit standards. Completed unit standards were then put together according to the qualification rules, to award a learner the qualification. Under the national qualification system, theory and practical unit standards were generally assessed in isolation from each other, and there was often no logical “thread” holding a qualification together in terms of a learner’s capability. Qualification achievement was based around meeting the qualification rules, without any consideration of the overall capabilities of the graduates on completion. Added to this, the assessment of unit standards was often fragmented and atomised, and even at a unit standard level there was little consideration given to a learner’s overall capabilities.
A new approach to learning and assessment
The assessment method for the old national qualifications in the agriculture sector consisted of the theoretical knowledge unit standards being assessed by contracted tutors, as part of classroom-based training days for each programme. The practical unit standards were assessed based on a trainee’s practical, on-farm performance, using a “work diary”. A trainee’s employer or workplace supervisor was their verifier. The work diary was filled in by the trainees and their verifier. The information in the work diary was used by the assessor, along with an assessment conversation with the trainee and the verifier, as evidence to make a judgement of competence against the practical unit standard.
The new integrated evidence portfolio approach was designed to be able to allow much more integration of practical tasks and theoretical knowledge, therefore enabling the assessors to make a more holistic assessment of trainee’s capabilities, based on the graduate outcomes of the qualification.
Development of the integrated evidence portfolio, and indeed the wider learning approach to the new qualifications, acknowledged some key adult learning principles.
Adult learners are different from children in their motivations, interests, values, attitudes, physical and mental abilities, and learning histories (Kennedy, 2003, cited in Westover, 2009, p. 435). Learning programmes designed for adult learners, such as those in a Vocational Education and Training (VET) context, are likely to be more effective if they recognise these differences and incorporate some key adult learning principles in their design.
Westover (2009) outlines ten important characteristics of adult learning:
- Learning is a process that lasts throughout the lifespan of most people
- Learners must be an active participant in the learning, not a passive recipient of information
- Learners must be responsible for their own learning
- The learning has an affective component as well as an intellectual component
- Adults learn by doing
- Problems and examples must be realistic and relevant to learners [italics in original]
- Adults relate their learning to what they already know
- An informal learning environment works best
- Variety is stimulating. A range of learning techniques is important
- Learning flourishes in a win-win, non-judgemental environment.
Both Kennedy (2003) and Knowles (1984, cited in Galbraith & Fouch, 2007, p. 36) have identified similar characteristics of adult learners, particularly those of practicality, relevancy, and the importance of life experiences.
The research project was done as an evaluative case study with two groups of dairy farm trainees, one based in the North Island and one in the South Island. The participants were dairy farm trainees who had achieved one of the old, national agriculture qualifications with Primary ITO, and had then enrolled in one of the new, New Zealand qualifications. They were asked to compare the assessment methods between the two qualifications. The trainees’ on-farm verifier (usually their employer or supervisor) and their Primary ITO assessor were also included in the research. Participants were interviewed individually and their thoughts on how the two assessment methods were captured.
Key findings of the research
The two key findings from the research project were: (a) as a result of the new approach to assessment, trainees were more able to apply the knowledge they had learned to workplace-based situations; and (b) that the new integrated assessment method resulted in a more authentic and robust assessment.
Application of knowledge to the workplace
The research found that the new integrated evidence portfolio assessment method enabled trainees to make strong connections between the theoretical knowledge learned in the classroom and the practical skills learned in the workplace. More opportunities to directly apply classroom knowledge to practical workplace skills not only improved the trainees’ performance of these practical skills but also resulted in a deeper understanding of the theoretical knowledge.
This finding reinforces the general principles of adult learning. Merrill (2002) proposed five principles of instruction which stated that learning is promoted when: (a) learners are engaged in solving real-world problems, (b) existing knowledge is activated as a foundation for new knowledge, (c) new knowledge is demonstrated to the learner, (d) new knowledge is applied by the learner, and (e) new knowledge is integrated into the learner’s world. Evidence from this research shows that the integrated evidence portfolio enabled effective learning by requiring trainees to: solve real world problems, have new knowledge demonstrated to them in practical ways, and apply and integrate this into their day-to-day farming lives.
This research demonstrated that trainees appeared to learn better and gain a deeper understanding of the theoretical concepts by applying them to real-world, problem solving contexts.
The data from the research shows that a key aspect of the improved, deeper learning of the trainees came from being able to utilise real-world data, information or experiences from their own workplace as part of the evidence for the assessment of their theoretical knowledge.
Examples of feedback on the new assessment method from the participants were:
“Theory and practical work more hand in hand because it is right there. They are completing them at the same time in the workbook” – assessor
“There seems to be a lot more cross-referencing between farm and class than in the past” – on-farm verifier
“It was more practical. It was more about what was going on on our farm” – trainee
Authentic and robust assessment
Another key finding from this research was that there was more “real world” or authentic evidence available to make assessment decisions. The study also found that the new integrated evidence portfolios encouraged and enabled more interaction and discussion between the trainees, verifiers and assessors as part of the assessment process. Improved interaction between participants, coupled with an increase in on-farm evidence, led to the conclusion that the assessment of the learning outcomes of the programmes were more authentic and robust than had been the case under the previous assessment system.
Gulikers, Bastiaens, & Kirschner (2004) investigated authentic assessment in depth, with their five dimensions of authentic assessment. The data from this research showed that assessment using the new integrated evidence portfolios was authentic and robust, and showed a strong alignment with those five dimensions, as shown in the table below.
Research findings compared to five dimensions of authentic assessment
Five authentic assessment dimensions
The assessment task – one that confronts students with activities that are also carried out in professional practice. They require students to integrate knowledge, skills and attitudes as professionals do.
Assessment of learning outcomes is based on performance of genuine workplace tasks, and underpinning theoretical knowledge needed for those tasks. Assessment is based on holistic, integrated evidence of trainees’ knowledge, skills and behaviour.
The physical context – an authentic assessment task should reflect the way knowledge, skills and attitudes will be used in professional practice.
Assessment evidence is gathered from actual workplaces in which trainees are employed.
The social context – authentic assessment should consider social processes that are present in real-life contexts. For example, collaboration.
Assessment is not contrived. It measures normal workplace behaviour and interactions such as questioning, collaboration, experimentation and problem-solving.
Assessment result or form – this relates to the quality, validity and fairness of the assessment.
Quality, validity and fairness of the evidence portfolio are determined by the evidence used, and the engagement, expertise and professionalism of those involved in making the assessment decision.
Criteria and standards – criteria are what the assessment is measuring (the outcome). Standards
Assessment is based on well-defined unit standards that reflect the learning outcomes of the programme, and thus the graduate outcomes of the qualification.
Authentic assessment is very valuable for learners. Gulikers et al. (2004) found that learners were stimulated to deeper learning when they perceived the assessment task to be authentic, while Timma (2007) found that authentic assessment practices influenced the way learners carried out their work. Learners “gained background knowledge and understanding about the reasons for work procedures being assessed” (p. 7). As workers, the learners valued “real life” assessment where they could demonstrate practical application of skills and knowledge through their actions and verbal responses to assessors.
Conclusion and application
These findings have important implications for people working with prison-based learners.
The literature review undertaken for the research; the data from the interviews with trainees, workplace verifiers and assessors; and the subsequent analysis of that data show that the key principle for workplace-based vocational training is context.
Ensuring that learning and assessment are embedded within real-world contexts – which in the case of vocational trainees mostly means their workplace – has huge benefits.
- Trainees will be more engaged with their learning, which makes the learning experience far richer and enjoyable for both learner and teacher
- Trainees are likely to obtain a deeper and clearer understanding of theoretical principles if that learning is in a practical workplace context
- Assessment will be more authentic and robust, which not only assists learners but gives greater confidence to employers that trainees have gained the skills, knowledge and behaviour that they require for successful workplace practices.
Ensuring that any education programmes that prisoners are engaged with are based around some sort of real-life context is critical. Whether it is industry training, literacy/numeracy programmes or other educational or rehab programmes, learners are more able to make sense of new information if they are able to relate it to everyday practical activities, and practice it in a natural, non-contrived manner. As far as assessment goes, utilising evidence that occurs naturally, in a workplace or daily life rather than a contrived situation – such as a written assessment – is likely to be a more authentic and robust assessment of a learner’s skills, knowledge or behaviour.
While a generalisation, evidence suggests that many learners who are engaged in more practical-based workplace activity are more likely to be predominantly kinaesthetic learners and to have had negative experiences with formal education. Re-engaging
such learners with a formal education process can be challenging for educators and stressful for the learners. We are doing both ourselves and our learners a favour if we can do this in a manner that is most likely to result in success for all concerned.
Couper, G (2016). Masters Thesis (unpublished). Massey University.
Galbraith, D.D., & Fouch, S.E. (2007). Principles of adult learning. Professional Safety, 52(9), 35-40.
Gulikers, J.M., Bastiaens, T.J., & Kirschner, P.A. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research & Development, 52(3), 67-86. doi:10.1007/BF02504676
Merrill, M.D. (2002). First principles of instruction. Educational Technology Research & Development, 50(3), 43-59. doi:10.1007/BF02505024
Timma, H. (2007). Learning, training and assessing on the job: What do workers think? Paper presented at the AVETRA conference, Evolution, Revolution or Status Quo? VET in New Contexts, Melbourne.
Westover, J.H. (2009). Lifelong learning: Effective adult learning strategies and implementation for working professionals. International Journal of Learning, 16(1), 435-443.