How to MOOC: Meaningful Assessment Through Real-World Problem Solving

So far in this series, we have looked at ways massive open online courses (MOOCs) have led educators and trainers to rethink how content is delivered and the role of social media in the corporate classroom. This article focuses on a topic that has historically been an albatross around the neck of training and development: assessment.

Assessment in corporate training is complicated by a couple of factors. First, there is a widespread misconception that exposure to information equals learning. The result has been an overabundance of objective testing methods that assess information recall but little else. This practice is probably responsible for the fact that employees retain only about 10 to 15 percent of what they learn in training sessions—information is easily forgotten; only when we apply that information does it become knowledge. The second complicating factor is even more troubling: many organizations don’t assess employee learning at all. In an interview with the Wall Street Journal last year, corporate training researcher Eduardo Salas noted that one of the biggest mistakes businesses make in training is failing to evaluate employee learning. If they do, he says, “they usually stop at the first level of evaluation—the reaction data. Companies think that if there is a positive reaction to the training, people will learn. But what we know is that the correlation is very week between reaction to training and actual learning.”

U.S. companies currently spend about $160 billion every year on training and development. Without evaluating what employees are actually learning, how can they determine whether that training is effective? What is the ROI on this substantial investment?

MOOCs have led educators to develop some new perspectives on assessment, and corporate training has a lot to learn from them. To start with, MOOCs are changing the place of assessment in courses in general. As Cathy Sandeen wrote in the May 2013 issue of Research and Practice in Assessment, “within the MOOC world, assessment is a central feature of design from the very beginning. In this new context, assessment is less about compliance than about supporting student learning outcomes and ultimately student success and attainment—directly in the center as it should be.”

Any type of standard assessment can be implemented in a MOOC. Objective assessments, like multiple-choice and fill-in-the-blank questions, can be administered and graded (with feedback optionally provided) using most learning management systems. Subjective assessments are also possible: writing assignments can be peer graded, oral presentations can be given via webinars, and groups can collaborate on projects in virtual workspaces. So the question becomes “What types of assessment can meaningfully support learning outcomes, success, and attainment in a training and development context?” The answer is the types that best lead to improved job performance.

A recent Forbes article identified the top 10 skills that organizations are looking for in new employees. It may surprise you to learn that hard skills such as computer programming didn’t even crack the top four. In fact, the four most in-demand skills are critical thinking, complex problem solving, judgment and decision-making, and active listening. These skills cannot be assessed using standard objective measures.

A host of new MOOCs are exploring the potential of the massive format to develop these skills and lead to meaningful outcomes using problem-based learning. In this model, students learn the content in a real-world environment and are assessed using real-world problems. These courses, which include the University of Washington’s “Introduction to Data Science” and the University of Virginia’s “Foundations of Business Strategy” maximize the advantages of the MOOC format by focusing on applied knowledge rather than information transfer, facilitating connections and collaborations among students, and crowdsourcing solutions to complex challenges. The courses are run in cooperation with Coursolve, a company that connects students with organizations that have problems to solve, so the students are working on real problems faced by real businesses. The organizations involved in this initiative range from entrepreneurs, small business owners, and non-profits to multinational giants like General Electric, Johnson & Johnson, and Walmart.

So, how can you best evaluate what your employees are learning?

As we mentioned, MOOCs can handle any type of assessment, and for some courses testing information recall using the testing module in your learning management system may be sufficient. But if you want your employees to translate that information into meaningful knowledge they will remember and can apply, consider using problem-based assessments.

There are several ways you can implement problem-based learning:

• For public MOOCs run on the Internet, you can design problems as case studies, simulations, or even games.
• For private MOOCs run on an intranet, you can use real challenges your company is facing and crowdsource talent from across the entire organization.
• You can have employees design or pick their own challenges to solve.
• You can consider joining with Coursolve to crowdsource your organization’s problem-solving on an even larger scale.

Unless you develop grading rubrics, problem-solving assessments may not result in standard scores like objective assessments do. But does it matter? The Internet and mobile technologies provide unlimited access to information so we no longer need to remember everything we learn. Assessment is only meaningful if it is tied to meaningful outcomes, and the goal of training is for employees to improve their core job skills, with many courses aimed at developing the skills in the Forbes top four. The effectiveness of these programs should be measured not by scores on a test, but by improved productivity and performance and their impact on the organization’s bottom line.

There are many other options for assessing learning, but the goal of this article was to get you thinking about the role of MOOCs and problem solving in training programs. In the final article in this series, we will take a look at different ways to recognize and reward MOOC achievements.

Copyright Bryant Nielson. All Rights Reserved.