Thursday, August 24, 2017

Why online proctoring doesn't work.

Proctoring exams is expensive and inconvenient.  

In an era when you can do almost everything online an exam candidate will be annoyed at having to make an exam reservation, take time off work, and travel across town to an exam center, just so they can be monitored by a real human.  The need for a physical exam location, computers, and a real human to monitor the candidate also increases the exam price, reducing the number of candidates, and reducing the profits of the exam vendor.

Obviously there is a business case for online proctoring, and there are plenty of companies that claim to have a solution that works, but in reality they will only catch an incompetent cheat, and a truly devious one will get away with a 100% score, courtesy of Google, or a friend in the next room.

A number of certification vendors now offer exams with online proctoring, but usually only for their introductory level exams, and not for the exams that lead to truly valuable certifications.  I guess those vendors understand the risk of cheating and dilution of the certification value, but decide the increased revenue is worth it.

What online proctoring solutions exist, and why don't they work ?

Shutting down other programs on the PC and locking the exam to full screen.

This sounds like a great idea, stopping users from running web browsers or chat clients ... until you realise that most people these days have multiple internet connected devices, and exam candidates could easily get a substitute to sit the exam, or look up answers on their phones, and enter them into the PC exam window.  FAIL.

Webcams and microphones and remote desktops.

The theory behind webcams and microphones is that they can monitor who exactly is sitting in front of the computer, to avoid substitutes, and watch for suspicious activity, like abnormal amounts of typing during a multiple choice exam.  Unfortunately webcams don't give a 360 degree field of view, so they can't check the room for additional computers or monitors, additional people, or trace computer cables.  The average highschool student would be able to setup a second monitor on the exam PC, so a friend sitting behind the webcam or in another room could view the screen, google for the answer, and signal to the test taker which answer to choose.  You need to secure the computer software and hardware, and monitor all activity in the room to avoid cheating.

Does that mean we can never stop cheating with online exams ?

Only a paranoid proctor can catch a devious cheat.  If the integrity of the exam is important, and any suspicion of cheating must be avoided, then the only solution is to use secured hardware, software, and real paranoid humans.  For low value screening exams, online proctoring may still be of use, but should never be considered to be 100% secure.  

One way to improve the integrity of online exams is to turn the exam into a timed race, where a candidate who KNOWS the answers and can answer quickly can get a higher score than someone who needs to search for an answer or communicate with another person, but this requires that the questions be based on memory recall, rather than reading comprehension or problem solving, and I am doing some research at Exam to build an online recruiting solution that implements this theory.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.