Massive open online courses (MOOCs) are a rising trend in corporate and workplace training. The courses are still fairly new, and many questions remain to be answered. Currently, one of the hottest topics is how to measure the success of a MOOC. Although once everything is up and running, the marginal cost associated with MOOCs can approach zero, they still require significant upfront investments of both time and money. Organizations interested in using MOOCs as part of their training programs need to have a clear idea of the benefits they will realize—preferably reflected in their bottom line.
In the previous post, I outlined the four-level model of evaluation developed by Donald Kirkpatrick. Here, we’ll explore how MOOCs and the data that comes out of them can be used to measure success at each of these four levels.
Level 1: Reaction
The reaction level refers to the way learners react to their experience of the course. Typically, this is assessed using “smiley sheets,” on which participants rate different aspects of the course on a Likert scale. This process usually happens immediately after a course is completed, and unfortunately it often represents the only evaluation that actually takes place.
Measuring success at this level is easily adopted into a MOOC, with the opportunity to collect much, much more data. In most MOOCs, the core content is delivered via video, and all of the learners’ interaction with the course materials takes place online. To increase engagement, many videos have quizzes embedded, and polls and surveys are common throughout the courses as well as during synchronous elements like live webinars. Rather than handing out a smiley sheet at the end, instructors can incorporate Level 1 evaluation into the MOOC by using polls and surveys to assess learners’ reactions in real time. In this way, not only will instructors be able to gather reaction data to individual elements of the course, but they will be able to use the data to make adjustments to the course if necessary.
Level 2: Learning
Learning in training programs is famously difficult to measure, but MOOCs provide a wealth of data that can be used to assess the progress of learning in real time. Depending on the platform and the data collection programs, learning management systems can collect massive amounts of data from MOOCs, including time on task, how learners interact with the course material, how many times they have to answer a question to get it right, and much, much more. The idea of combining big data with learning analytics is still very new, but already it is providing a powerful way to assess learning and engagement that wasn’t available before.
Level 3: Behavior
For learners and organizations, this is the most important individual level of measurement—the ability of learners to apply the new knowledge, skills, and attitudes they acquire in courses to their work. In the end it is the behavioral outcomes, not satisfaction with the training or how long it takes to work through a course, which will determine the impact of the program on an organization’s bottom line. This seem like the most opaque category of outcomes, but many newer MOOCs provide an excellent way to directly measure how well learning translates into behavior—by using real problems and projects as course assessments.
Many instructor-led training programs don’t have much in the way of evaluation at all, and even traditional elearning relies far too heavily on multiple-choice assessment. But MOOCs, in which huge numbers of learners share resources, hold virtual discussions, and collaborate on projects, provide an ideal environment for incorporating the desired new behaviors directly into the training. This practice blurs the line between “training” and “work” and allows employees to put their newly acquired knowledge, skills, and attitudes directly into practice. This can help foster a learning mindset throughout the organization and also give learners a chance to practice their new skills, like decision making, in a lower-pressure, but real-world, environment such as a simulation. See here for a more in-depth look at how some companies are using MOOCs to solve real business problems.
Level 4: Results
Here is where all of the standard metrics—reduced turnover, increased job satisfaction, increased productivity, increased customer satisfaction, and so on—come into play. If the training program has proved successful at the other levels (particularly Level 3: Behavior), then Level 4 should emerge quite nicely. Remember that in order to achieve valid results, organizations need to decide in advance what metrics are tied to each course as well as how those metrics will be measured—in hard numbers if possible.
At this fourth level, there are two final things that are important to keep in mind. First, training is an investment, not a cost. What’s important is the ROI an organization will realize as a result of the training program, and for some metrics (like the bottom line) this return may take a longer to show up than, for example, the reaction data on smiley sheets, but the costs of not training employees are infinitely higher than the investment in training them. Second, measuring the ROI of training programs in general is very new, and thus far it is virtually untested in MOOCs. So, if the first numbers don’t look as good as hoped, instructors can use them as guides for how to improve the courses in the future. This points to an additional advantage of MOOCs—because the courses are broken up into bite-sized pieces, if something isn’t working, it’s easy to replace just that portion of the course and immediately put the new material in the hands of all learners.
MOOCs are still very new players in the corporate training environment. The goal of these two articles has been to explore a way to meaningfully assess the value of training MOOCs, a topic that will have increased relevance in the near future as more companies adopt this new format and the evaluation tools improve.
Copyright 2017 Bryant Nielson. All Rights Reserved.
|
Bryant Nielson – Managing Director of CapitalWave Inc.– offers 25+ years of training and talent management helping executives, business owners, and top performing sales executives in taking the leap from the ordinary to extraordinary. Being a big believer in Technology Enabled Learning, Bryant seeks to create awareness, motivate adoption and engage organizations and people in the changing business of education. Bryant is a entrepreneur, trainer, and strategic training adviser for many organizations. Bryant’s business career has been based on his results-oriented style of empowering the individual. Learn more about Bryant at LinkedIn: www.linkedin.com/in/bryantnielson |
 |
