In my talks with the heads of training departments and other company leaders, I have found that many have a high level of interest in massive open online courses (MOOCs), but that they feel there are still too many unknowns. That’s why several of my recent posts have addressed resistance to MOOCs, such as last week’s article on the not insignificant issue of barriers to organizational change.

Here, I propose a solution for training departments that are interested in seeing what MOOCs can do, but aren’t yet totally convinced, or perhaps haven’t been able to get the necessary buy-in: A/B testing. This article explores what A/B testing is, why it is valuable, and how to apply it to your training programs.

What is A/B testing?

Buffer’s Kevan Lee has a great, simple definition for A/B testing: “an A/B test is a way to measure two versions of something to see which is more successful.” Essentially, it is running an experiment with two groups to see which group has the best results.

A/B testing is done all of the time, particularly in marketing and web development. For example, marketers might test two versions of a button to see which one generates the most clicks, or two versions of web copy to see which one results in website visitors staying on the site longer. The idea has even been used before, to some extent, with MOOCs — a few studies have compared results between students taking a course the traditional way (i.e., in person, via lecture) and students taking the same course at the same time, but as a MOOC.

Why use A/B testing for training?

The goal of A/B testing is to see which of two things works better. Many training departments today are struggling with how to provide more training, faster, and on a reduced budget. If this describes your situation, aren’t you interested in learning what methods really lead to the best training results for your company?

Here are some of the main benefits of A/B testing a training MOOC:

• It’s a good introductory step. Your organization may not be ready to move your entire training program over to a MOOC. A/B testing is a first step that will allow you to see what you think of the new format, while also gathering data that will help you make a decision for the future.
• It’s objective. A/B testing takes the guesswork out of the equation. It allows you to objectively test two types of training and measure the results. Based on those results, you can make the best decision for your organization.
• You gain valuable insight into what works. Measuring the success of training is tricky. Depending on the metrics you use, the results may or may not be valid or generalizable to on-the-job performance. With A/B testing, you can gain insight into what works and what doesn’t, which will help you refine your training programs going forward.

How do you A/B test a MOOC?

In regular A/B testing, the goal is to change only one variable and keep everything else constant. For example, you may release two versions of a website that are identical in every way except that one has red buttons and one has green buttons. This is not strictly possible for a testing a MOOC versus instructor-led training because the formats are so drastically different and it may not be possible to use exactly the same content in each case.

But you can still conduct a test using the method outlined below:

1. Pick the right course. Ideally you would test the exact same course using both methods, but that’s not always feasible. Instead, you can pick a course that is fairly standard, such as a course on business communication.
2. Choose your training metrics. Retention, attitudes toward training, business performance metrics — these are all measurements you can use to judge training effectiveness. MOOCs generate a lot of data, but remember that whatever metrics you choose will also need to be measured for ILT. Need help? Check out these top 10 training metrics.
3. Divide your learners into groups. The groups should be decided randomly and should contain at least 100 people each. In our experience, the breakeven point for the cost of MOOCs compared to ILT is a two-day training program for 100 people. So don’t test a MOOC with just 20 people — it won’t be cost-effective and it will be less likely to yield statistically significant results.
4. Run the courses at the same time. If possible, run the courses at the same to decrease the chances of outside factors interfering with the results.
5. Analyze the data. As they say, the proof is in the pudding. When your A/B test has completed, analyze your data to see which format comes out ahead.

MOOCs are new. They require a different way of thinking about training and about technology than most organizations are used to. They may not be right for your organization. But on the other hand, they might be exactly what your organization needs to deliver the volume and quantity of training that is required. But you’ll never know until you give it a try.

Contact me to learn more about MOOCs and for help setting up an A/B test for your training program.

Copyright 2015 Bryant Nielson. All Rights Reserved.

Bryant Nielson – Managing Director of CapitalWave Inc.– Being a big believer in Technology Enabled Learning, Bryant seeks to create awareness, motivate adoption and engage organizations and people in the changing business of education. Bryant is a entrepreneur, trainer, and strategic training adviser for many organizations. Bryant’s business career has been based on his results-oriented style of empowering the individual. Learn more about Bryant at LinkedIn: www.linkedin.com/in/bryantnielson