Now that you’ve decided to use simulations in your programs, let’s look at some best practices for evaluating simulation results.
A well-designed simulation will only be effective if you are able to evaluate the results – and pass those results on to the participants. As we’ve discussed, immediate feedback is a benefit of simulations, so the evaluation of final outcomes should be fairly immediate so that participants can quickly apply what they’ve learned. Let’s discuss some ways to create simulation evaluations in a way that makes them useful to both the organization and the participants.
The first step in creating effective simulation evaluations is to look closely at the delivery method. Obviously if the simulation is a complex, computer-based operation, then the programming should also deliver an evaluation in an immediate context. For example, a flight simulator will create a plane crash if the pilot has made grave mistakes. Action-based simulations, like putting out a fire or building a piece of furniture should not only be based on the quality of the final outcome but also on the time it took to reach the outcome. If the fire has been extinguished, how much time did it take and how much structural damage was done? Or, if the chair has been assembled, how long did it take and will it collapse when someone sits in it? Case study simulations should be based on the outcomes and, like all of the other simulations, on the consequences of wrong actions. We will examine this in just a moment. Finally, if a group is involved, be sure to evaluate how well the group worked together as well as the contributions of individual members.
For any simulation, whether complex or not, take the time to list the desired outcomes. For example, a financial simulation could have outcome levels, such as cash savings of $100,000, $75,000, and so on. An HR-based simulation could have outcomes of successfully delivering permanent pay cut notices with a minimum of attrition. No matter what the topic of the simulation, the evaluation has to start with the desired outcomes.
Reaching a successful outcome is one aspect of simulation, but participants should also know if they have taken the preferred steps for those outcomes. The preferred steps should coincide with applicable laws, natural phenomenon, organizational procedures, and even organizational culture. For instance, the HR simulation may end with a low attrition number but what happens if the participant tells simulated employees that their pay will rise back to its original point within a few months, when the cut was permanent?
Not only is it necessary to examine the preferred steps for evaluation, it is also necessary to look at the consequences for wrong actions. One way to design this part of the evaluation is through the use of a decision tree that maps out the right steps, the wrong steps, and the consequences. Consequences for wrong steps are a big part of simulation, because they help the participants learn and apply knowledge to the situation. With that in mind, remember to explain consequences in terms that are correlated with the simulation, such as lost dollars, lost time, or potential attrition rates. The ability to compare right steps with wrong steps using the same units is invaluable in application. Along these lines, though, be sure to have moderators point out correct thought processes even if the eventual step is incorrect. This may be especially true in group simulations, where some group members wanted to take the correct step or process.
Finally, create a matrix or rubric that shows the criterion for the evaluation so that it is useful to both the moderator and the participants. For example, if a participant or group chooses a right step but makes errors along the way, their partial credit should reflect this and point out what was correct in their thought processes. In addition, weights in the evaluation should coincide with weights in the real world. In other words, a loss of dollars that causes an organizational bankruptcy should be weighted much heavier than a loss of dollars that barely causes a shudder. Both are wrong, but, as in the real world, sometimes the wrong choices carry degrees of consequence. Keep in mind that choices that are contrary to organizational culture or applicable law should be heavily weighted, as well.
Here is one final tip on evaluation: if the simulation has multiple parts, be sure to create an evaluation for each part.
