Showing posts with label Kirkpatrick. Show all posts
Showing posts with label Kirkpatrick. Show all posts

Saturday, March 6, 2010

Obsessing over Assessment

The law of attraction is at it again. Testing and assessment is running through my mind and it seems to be running through my life as well. There is a certification project currently being implemented in my company that involves training and testing. I am taking a course on Inquiry and Measurement that also involves testing. And, during this week’s #lrnchat, I was drawn in by a discussion thread in which the pros and cons (well, mostly cons) of assessment were being discussed. Why am I so preoccupied with Level 2 of Kirkpatrick’s Four Levels of Evaluation? It is mostly because of the certification project at work.

Certification is a term that gets thrown around a little too loosely in training departments these days. Put someone through a course and a post-test and BANG, you are certified – or worse, you are not. It is a tricky thing to put together a certification process that is valid and reliable. Very few companies can stand to wait out validation process, so they jump right in and begin training and “certifying” people.

Certifications do not guarantee that the person being certified has learned more than he or she would in a regular training program, but business leaders often feel it is operationally necessary to validate a level of knowledge or skill required to meet goals and targets. If certification is needed or required, here are some things to consider for the assessment process:

Test items should be directly related to learning objectives, which should be directly derived from performance requirements. This may seem obvious but I have seen many tests that have included filler material alongside valid questions.

Test only on important items, not obscure ones. It is not necessary to test someone on small details unless they are critical. Very few corporate employees are doing life-saving work that needs to be tested at a granular level.

Test items should be straight-forward. Don’t try to be tricky. What is the point? It only serves to confuse the learner and adds no value to the assessment.

Choose question formats that make sense for the level of learning you need to assess.  Multiple choice questions are commonly used on knowledge tests because they are easy to score and easy to tie to outcomes – but they are not always easy to write. Good multiple choice questions will have a clear premise in the stem of the question, a correct answer, and reasonable alternative choices. There should not be any throw away responses or convoluted choices such as “a and b, but not c” or “a and c” only. If your test item has more than one correct answer, then rethink the question format. Consider short answer questions or a matching column.

If possible, use randomized test questions. Most learning management systems have this capability. They allow you to create a bank of questions that can be drawn upon at random so that test-takers will be deterred from sharing answers. But don’t make the bank of test questions so large that everyone feels like they are taking a completely different test.

Pilot your test. This is the hard part, because it takes time and patience. You need to let a few people complete the learning experience and take the test to give you the opportunity to analyze the questions. You will want to take a second look at questions that everyone got right, or everyone got wrong, or questions for which many people chose the same incorrect answer.

Create rubrics for skills assessments.  Skills assessment usually requires direct observation.  It is important that all of your assessors are using the same criteria and weights when judging performance.  Validate the process by having multiple assessors review the same performance.  If they are more than a few points off from each other, either redesign the rubric or re-train your assessors.

Saturday, October 17, 2009

Five Examples of Learning Metrics that Matter




This week I participated in a discussion on learning metrics with my new friends at #lrnchat. What a pleasure it is to be part of these weekly events. Where else can you say things like, “if we start analyzing the problem using the Six Boxes® model based on Thomas Gilbert’s theory, we are sure to make some progress” without people running away to see if someone refilled the punch bowl?

The questions that were tossed around this week were:
  • Are learning metrics different from business metrics?
  • What are you measuring in your organization?
  • How do you tie organizational learning to business performance outcomes?
  • What else besides metrics do you use to show impact?

I was excited and a little bit surprised as I watched the responses roll by. I get so caught up in what I’m doing at work that I lose sight of what is going on in the rest of the world. Some people seemed to be struggling with measurement basics, while others seemed to have a strong grasp on the topic. This got me thinking about what we are doing at my organization, which probably falls somewhere in the middle.


I thought a lot about this discussion after it was over. Really, the heart of the matter is: What value do we as learning professionals bring to the organization? How do we show it? The short answer is that we do this by helping business leaders meet their goals.


I have been fortunate throughout my career to have been involved in a number of projects that made a huge impact on the business. But I can also recall times when it was difficult to see connections between the work that I was doing and the impact it was having on the organization. These are the times when we all fall back on measures such as, “butts in seats,” level one evaluations, and elearning completion rates. These are important feedback indicators for the training department, but they usually don’t mean very much to business leaders. That being said, I can think of examples in which these measures alone have been important to the business. What it comes down to is that sometimes the measures associated with learning success will be obvious and glamorous, and other times they will not. Here are a few examples of both from my experience:
  1. Sales Training is one of those cases where the metrics are obvious and easy to align to the business. We run a class on how to sell against our competitors effectively. The natural metric associated with this is the number of competitive takeaways. We have clear evidence that sales reps who take this class have a higher rate of competitive takeaways than the general rep population.


  2. Equipment Service Training is another example with obvious metrics. Well-trained service reps fix problems more quickly and are able to tackle more service calls in less time.


  3. Operations Training can have a variety of metrics. We ran a class on how to write work instructions for a part of the business that was very procedurally oriented. Every time they had people leave the department due to promotions or turnover, processes broke down. Our metric was the existence of well-written work instructions in the departments that participated in the training. This helped the business leader keep continuity when there was turnover.


  4. Compliance Training metrics are not very glamorous. This is a case where counting up completions works to serve the measurement need. Our legal department wants to reduce or eliminate ethics violations, or failing that, be able to prove the company did its part in making employees aware of their responsibilities and the consequences. We make sure everyone goes through our business ethics course and provide a report shows that this has been done.


  5. Soft Skills Training metrics are of course the fuzziest area. Yes we conduct level three evaluations to show behavioral change, but most people don’t get too excited over these. However, our Chief Human Resources Officer is very interested in employee engagement. She knows that offering soft skills training helps people with personal development which is important to engagement. She also knows that managers and supervisors who use what they have learned in soft skills courses are going to receive higher engagement scores. So yes, we count “butts in seats” for these courses.
The point I’m trying to make with these examples is that the metrics have to match the need.  When we help our organization’s leaders meet their goals by identifying and addressing the learning component of their business problem, we are truly adding value.