Ideas on How to Evaluate Your Training Programs
02/08/22 22:50
Using Learning-Transfer-Evaluation Model (LTEM)
The Problem
Reports show that training evaluation isn’t effectively carried out in most companies. Several reasons have been attributed to it:
1) lack of resources
2) inadequate evaluation skills
3) time constraint
4) lack of management support
5) not using appropriate assessments to measure learning
A more problematic practice is that most companies rely on self-report evaluation surveys to measure the effectiveness of their trainings (the most used evaluation model is the Kirkpatrick model, and most companies evaluate level 1-Reaction, level 2-Learning, & even level 3-Behavior in these self-report surveys)! Obviously, data that are collected from these surveys aren’t accurate enough to tell you about the effectiveness of training programs.
Why Evaluate
First, if you know how your trainings are, you can improve the way you design and deliver your trainings. Second, evaluation can provide data on cost-effectiveness of the training. And last, it can give you useful information to reinforce transfer of learning. There are many evaluation models in industry, and you can choose any as you see fit according to your needs. I propose the learning-transfer evaluation model (LTEM) given my long-term experience with the Kirkpatrick model and the way it was used for training evaluation.
Using LTEM
To use LTEM see the illustration that sums up the entire process, which I call result-driven design, and read the details below it. When I first saw LTEM, my immediate thought was this is a great tool for training design too. In other words, if I know what exactly I’m going to measure in the end, I’ll work towards that in my design. Most importantly, data that are gathered from assessments can help change the stakeholders’ approach to training (a very common approach, which is “information dissemination to build skills” ).
1. Introduce LTEM to your stakeholders, in particular SMEs. They should know how the model works and why it should be used as a reference in training design. You can even use LTEM to guide your discussions on needs analysis. You can identify knowledge, decision making, and skills gaps in performance.
2. By digging deeper into the gaps, decide which tiers are going to be measured once you complete your needs analysis.
3. Work backwards by breaking down tier 6-Task Competence into sub-tasks, decision making, and knowledge required to perform the task(s) accurately and efficiently.
4. Then, determine the target competencies and performance objectives based on the tiers that you want to measure (you won’t need to write objectives for knowledge – you are merely focusing on ‘performance’ )
5. Now decide how you are going to assess competency areas targeted by training. Your assessment methods and techniques are key in providing useful data on training. Aligned with each key task and performance objective, develop your assessments (i.e., criterion-referenced assessment). For example, if teamwork is one of the target competencies, make sure to assess learners for a team-based task. If your training is online, write scenario-based questions. If you need to assess the learner’s knowledge as well, this should be assessed in contextual real-world conditions (i.e., criterion-referenced). Most importantly, for skill assessment, design a blended training and use demonstration of task(s) in-person. For this, develop robust checklists or rubrics for assessment by an expert (this should be done in collaboration with your SMEs).
6. The last step is developing content and training methods (i.e., training strategies). For example, if teamwork is part of your training target competencies, you can use crew resource management (CRM) training strategy in simulation-based training to optimize teamwork and train team members to use all available resources. Although this strategy is known for aviation, it is applicable to other industries that require teamwork. Crew resource management (CRM) uses a range of behaviors from communication, decision making, assertiveness, leadership to situational awareness and adaptability. Make sure to consider tier 7-Transfer and include job-aids and after-training support toolkits. Here’s a checklist to design CRM training.
7. If you intend to measure higher tiers, such as tier 7-Transfer or tier 8-Effects of Transfer (that would be the impact), you’ll need to create a logic model and plan surveys, vignettes, or focus groups as needed.
The Problem
Reports show that training evaluation isn’t effectively carried out in most companies. Several reasons have been attributed to it:
1) lack of resources
2) inadequate evaluation skills
3) time constraint
4) lack of management support
5) not using appropriate assessments to measure learning
A more problematic practice is that most companies rely on self-report evaluation surveys to measure the effectiveness of their trainings (the most used evaluation model is the Kirkpatrick model, and most companies evaluate level 1-Reaction, level 2-Learning, & even level 3-Behavior in these self-report surveys)! Obviously, data that are collected from these surveys aren’t accurate enough to tell you about the effectiveness of training programs.
Why Evaluate
First, if you know how your trainings are, you can improve the way you design and deliver your trainings. Second, evaluation can provide data on cost-effectiveness of the training. And last, it can give you useful information to reinforce transfer of learning. There are many evaluation models in industry, and you can choose any as you see fit according to your needs. I propose the learning-transfer evaluation model (LTEM) given my long-term experience with the Kirkpatrick model and the way it was used for training evaluation.
Using LTEM
To use LTEM see the illustration that sums up the entire process, which I call result-driven design, and read the details below it. When I first saw LTEM, my immediate thought was this is a great tool for training design too. In other words, if I know what exactly I’m going to measure in the end, I’ll work towards that in my design. Most importantly, data that are gathered from assessments can help change the stakeholders’ approach to training (a very common approach, which is “information dissemination to build skills” ).
1. Introduce LTEM to your stakeholders, in particular SMEs. They should know how the model works and why it should be used as a reference in training design. You can even use LTEM to guide your discussions on needs analysis. You can identify knowledge, decision making, and skills gaps in performance.
2. By digging deeper into the gaps, decide which tiers are going to be measured once you complete your needs analysis.
3. Work backwards by breaking down tier 6-Task Competence into sub-tasks, decision making, and knowledge required to perform the task(s) accurately and efficiently.
4. Then, determine the target competencies and performance objectives based on the tiers that you want to measure (you won’t need to write objectives for knowledge – you are merely focusing on ‘performance’ )
5. Now decide how you are going to assess competency areas targeted by training. Your assessment methods and techniques are key in providing useful data on training. Aligned with each key task and performance objective, develop your assessments (i.e., criterion-referenced assessment). For example, if teamwork is one of the target competencies, make sure to assess learners for a team-based task. If your training is online, write scenario-based questions. If you need to assess the learner’s knowledge as well, this should be assessed in contextual real-world conditions (i.e., criterion-referenced). Most importantly, for skill assessment, design a blended training and use demonstration of task(s) in-person. For this, develop robust checklists or rubrics for assessment by an expert (this should be done in collaboration with your SMEs).
6. The last step is developing content and training methods (i.e., training strategies). For example, if teamwork is part of your training target competencies, you can use crew resource management (CRM) training strategy in simulation-based training to optimize teamwork and train team members to use all available resources. Although this strategy is known for aviation, it is applicable to other industries that require teamwork. Crew resource management (CRM) uses a range of behaviors from communication, decision making, assertiveness, leadership to situational awareness and adaptability. Make sure to consider tier 7-Transfer and include job-aids and after-training support toolkits. Here’s a checklist to design CRM training.
7. If you intend to measure higher tiers, such as tier 7-Transfer or tier 8-Effects of Transfer (that would be the impact), you’ll need to create a logic model and plan surveys, vignettes, or focus groups as needed.