Measuring Success (ROI) of a Training MOOC, Part 1

Massive open online courses (MOOCs) have been around for almost four years, and in that time, a large amount of data about the courses has been generated. Now, researchers are starting to probe what kinds of students take MOOCs, why they take them, and what factors contribute to individual student success. These are important, no doubt, particularly in higher education, but companies interested in using MOOCs for training and development are also interested in the success of the courses as a whole—not just for individual students, but for organizations overall. This is an especially interesting issue for a couple of reasons: first, there is no consensus on how to measure the success of a MOOC in any environment, and second, companies are notoriously bad at measuring the returns on investment of their training programs in general.

Over the course of of two articles, I will explore what constitutes meaningful measurement of training and how this measurement can be applied to MOOCs. 

How Is MOOC Success Measured?

This question has generated a significant amount of controversy. The first metric to be considered was the number of students completing the courses with passing grades (usually defined as 70 percent or better). Depending how you look at it, this metric leads to either an excellent or a terrible conclusion. If you consider the actual numbers of students, MOOCs have successfully educated millions of people worldwide, and instructors have been able to reach more students in a single class than they normally could in an entire career. But, if you instead consider the percentage of students who complete MOOCs, the picture is quite different—fewer than 10 percent of students who sign up for MOOCs stick it out until the end.

But this is not really a valid way to measure success. As I described in an earlier post, many people sign up for MOOCs without ever intending to finish them, so completion rates don’t really tell us much. Raw numbers of students may be better, but without knowing what students were required to do to complete a course, they still don’t really tell us anything meaningful.

So, what would be a meaningful measure of MOOC success? The current move in education, which is also valid for training and development, is to measure a MOOC according to its value. Then the question becomes, “What constitutes value?” To determine a MOOC’s value for education, Owen Youngman of Northwestern University suggests going straight to the source and asking the students. But for organizations, where training departments often need to constantly justify their existence and their expenses, we need to develop a more concrete way to measure value.

How Are Training Programs Evaluated?

Identifying a meaningful way to measure the value of MOOCs in organizations is complicated by the difficulty of measuring the ROI of training programs in general. The most widely used (at least in theory) method of evaluating training programs is the four-level model developed by Donald Kirkpatrick. According to this model, the four levels are:

1. Reaction – The way learners react to their experience of the course
2. Learning – The new knowledge, skills, and attitudes gained from the course
3. Behavior – How well the new knowledge, skills, and attitudes are applied on the job
4. Results – Reduced turnover, improved job performance and satisfaction, improved organizational performance, etc.

I mentioned that this method is widely used—at least in theory. This is because in reality evaluation usually stops after Level 1: Reaction. Employees are asked to rate their satisfaction with various aspects of a course (often using what are known as “smiley sheets”), and then the evaluation is over. Despite the fact that the goal of training for employees is to learn knowledge, skills, and attitudes they can apply on the job (Level 3), and for organizations the goal is demonstrable improvements in the bottom line (Level 4), these outcomes are rarely measured. According to a 2006 report by Bersin & Associates, organizations feel that the number one most important thing to measure is how training programs impact employees’ jobs and the business, but less than 10 percent of organizations actually measure these outcomes. Clearly there is ample room for improvement.

The key to measuring the effect of training programs on higher-level outcomes is twofold:

• First, for each course or program, create a list of agreed-upon metrics that will be used to measure success.
• Second, decide how each metric will be determined. As much as possible, tie each metric to a specific number or dollar value. For example, customer satisfaction, productivity, and turnover and retention rates can easily be converted into hard values. For less tangible metrics, like decision making, change leadership, and communication, use a combination of hard values and anecdotal evidence, as advocated by Jack Philips of the ROI Institute.

In this article, we’ve examined some of the challenges associated with measuring the success of MOOCs and training programs in general. In the next post, I’ll explore some specific recommendations for how the tools and principles behind MOOCs can be used to assess the impact of training on the different levels in Kirkpatrick’s model.

Copyright 2017 Bryant Nielson. All Rights Reserved.

Bryant Nielson – Managing Director of CapitalWave Inc.– offers 25+ years of training and talent management helping executives, business owners, and top performing sales executives in taking the leap from the ordinary to extraordinary. Being a big believer in Technology Enabled Learning, Bryant seeks to create awareness, motivate adoption and engage organizations and people in the changing business of education. Bryant is a entrepreneur, trainer, and strategic training adviser for many organizations. Bryant’s business career has been based on his results-oriented style of empowering the individual. Learn more about Bryant at LinkedIn: www.linkedin.com/in/bryantnielson