While researching a Brandon Hall Group client’s question about assessments, I was surprised by the original development date of the Kirkpatrick Levels: 1959. In 1959 the following were true:
- Average Cost of new house $12,400.00
- Cost of a gallon of Gas 25 cents
- Loaf of Bread 20 cents
- Alaska becomes the 49th State of the United States
- Hawaii becomes the 50th State of the United States
- It would be 8 years until the first Super Bowl.
Kirkpatrick’s levels were useful when measuring the effectiveness of workers in a manufacturing environment. The Level 1 “smile sheets” worked in any situation. Today, in 2014, Brandon Hall Group recently surveyed L&D leaders across the country and found that they are, for the most part, still on Level 1 when it comes to using Kirkpatrick’s 1-4 scale.
That raises an important question. Do the Kirkpatrick Levels measure anything worth measuring anymore? Today very few people who go through a training program work in a manufacturing plant at a physically demanding job. Most of you reading this are knowledge workers. We learn in ways that were unimaginable in 1959. Yet the levels were developed within a 1959 context for training.
My friend Dean Spitzer is the author of a classic book about the need to change not only what we measure but also how we measure the effectiveness of learning. His book, “Transforming Performance Measurement: Rethinking the Way We Measure and Drive Organizational Success,” explains why performance measurement “should be less about calculations and analysis and more about the crucial social factors that determine how well the measurements get used.” His “”socialization of measurement”” process focuses on learning and improvement from measurement, and on the importance of asking questions like these:
- How well do our measures reflect our business model?
- How successfully are they driving our strategy?
- What should we be measuring and not measuring?
- Are the right people having the right measurement discussions?
In this new context, the smile sheets at Kirkpatrick Level 1 hardly seem worth the paper upon which they were once printed. Even Level 4, which was rarely if ever measured, seems dissonant with the idea that you need to measure what your business is all about, and what will bring you closer to your strategy and corporate goals. Letting the L&D department measure learning programs with smile sheets may make the instructors feel good, but it will have little to no value when you need to measure what really counts.
So perhaps it’s time to jettison Kirkpatrick and have it replaced with a more robust and dynamic model like the one in Dean’s book. The L&D organization touches almost everyone in the company. That puts them in the right place with the wrong tool. Given the right tools, the L&D organization would be a good choice for carrying out the measurements that really count.
At the end of the day, we manage what we can measure, and get what we measure. If you measure the wrong things, you get the wrong things. You take your company further away from the performance and productivity demanded by your strategic goals. Unless of course all you want are smiling happy learners who enjoyed their time in class.
What do you think? Does our old friend Kirkpatrick and his levels still have any relevance to the demands placed on today’s L&D organizations? Or are the levels an artifact of a previous era, in which most people worked with their hands, and gasoline was 25 cents a gallon.
-David Grebow, Principal Learning Analyst, Brandon Hall Group