Original Publication: Customer Management Insight - August 2008
View Online
I’d like to see our learning assessments modeled more after the driver’s exam. Think back to when you first got your driver’s license. Yes, that’s a long time ago for me, as well, but there were likely two parts to your driver’s exam — a written test and a performance test.
On the written test, you had to answer questions about traffic laws and safety rules. On the performance test, you had to actually drive a car. The examiner rode in the car with you and determined whether you could stay within your lane, stop at traffic lights, make turns without hitting the curb, etc.
How would you feel if new drivers today only had to pass the written exam and didn’t have to show they could actually drive a car? Would you be afraid to be on the road with these new drivers?
Often, we rely exclusively on written tests to determine learner ability because written tests are easy to administer. But can a multiple choice, fill-in-the-blank or true/false question truly measure a person’s ability to perform the following skills?
- A call center representative calmly responding to an irate customer complaining about a charge on a bill.
- A plant worker properly disabling equipment to avoid injury while cleaning or repairing the equipment.
- A technician installing a software upgrade on servers that could bring down your company’s network.
The challenge
Perhaps the biggest challenge to implementing performance assessments is time. Performance assessments can be time-consuming if you need to observe each individual perform a task from start to finish. With some tasks, you can review the final product, allowing more learners to take the assessment simultaneously. Whether you are testing in groups or individually, convincing a client to provide time for learner demonstration by either removing content or adding time to a course is challenging for consultants — and even inhouse instructional providers. However, it’s worth the effort.
The following ideas can make the process of developing the performance assessment more effective and rewarding.
Enlist your expert agents
As you probably won’t have time to test everything in the course, your subject-matter experts (SMEs) can help identify the most critical tasks to include in the assessment. I find that SMEs embrace the concept of performance assessments, and they can adeptly pinpoint the important tasks. Your SMEs can also help set passing scores and determine if learners should get more than one try to pass the exam.
Coach your SMEs on how to make the assessment realistic. If learners can use manuals, online help and other tools on the job, they should be able to use the tools during the assessment.
Develop a recording and scoring tool
Checklists and rating scales are the most common tools for recording and scoring learners’ performance.
Checklist
The person who administers driving tests uses a checklist that details what’s being observed — turns head and watches rear of vehicle while backing vehicle straight, smooth and slow for 50 feet — with a place to record the result.
Consider the following when developing your checklist:
- What specifically should the evaluator observe or check? What are the criteria for success?
- Does the learner need to perform the tasks in a specific order?
- What are common learner errors?
- Does the evaluator need room to write comments?
Rating Scales
Rating scales have the advantage of allowing evaluators to indicate a quality or level of performance. However, consistency in scoring is often a problem with rating scales. For instance, two call center supervisors can listen to the same representative’s call and give two different ratings — one scores the call a 3 and the other scores the call a 4. Why do the ratings differ? There’s no clear definition of a 1, 2, 3, 4 or 5 rating. If you decide to use a rating scale, provide enough details for each rating to ensure consistency across raters.
Figure 2 is a rating scale example for a new-hire program for a cellular phone company. In this sample scenario, new customers receiving their first bill often see two charges — one charge is for the time the customer signs up for service until the bill run date and another charge for next month’s service, which is paid in advance. Confused and angered when seeing two charges on their bill, customers call for an explanation. The levels of employee response are then detailed.
Notice how each rating — 1 through 5 — contains an example of what the representative (learner) says to the customer. Providing details of the responses that would merit each rating level improves evaluator consistency and establishes performance standards for the learner.
The bottom portion of the evaluation form shows the screens the learner uses to answer the call.
You want to make sure the learner is using the correct and most efficient screens to access customer information. There are two ways to observe the learner’s keystrokes. One option is for someone to watch and write down the keystrokes. The other option is to purchase software that records the keystrokes.
You also want to define ratings for how to open the call, steps for greeting customers and verifying account information, and how to close the call by asking, “Is there anything else I can help you with today?”
Combining all of these pieces of information about the new-hire’s response to the scenario provides a broad view of his or her learning and skills that simply wouldn’t be available if the only testing method were a written exam.
Figure 1: Evaluator Checklist 
Figure 2: Sample Rating Scale for a New-Hire Program
No Surprises
Although written exams are easy to administer and performance exams present challenges, there are certain outcomes that require actual, hands-on learner demonstration. In order for learners to obtain a license from your class, toss them the keys and ask them to drive.
Debbie Biggs is an instructional design consultant. dbiggs06@comcast.net
