Monday, October 01, 2012

Towards Better Test Taking Software

Almost thirty years ago, when I was teaching at Tulane Business School, I had a project to develop software to let students take exams, and professors grade them, on computer—the name of the product was going to be "Electric Blue Book." Unfortunately, it never got to completion--something that has happened with a number of my projects. But many years later, other people did create such software. The version I'm familiar with, because my law school uses it, is Computest.

What is depressing about the software is that all of the ingenuity has gone, not into making it easier to take and grade exams, but into making it harder to cheat. From the standpoint of the user, the software is clearly inferior to what I designed but never got programmed. For example ...

In my version, the student would see a page listing questions and showing, for each question, whether he had done it and whether, if so, he wanted to go back to it if there was time. In my version, the professor would be able to have the software feed him all the answers to question one in random order, then all the answers to question two, and so on. Doing it that way makes it easier to remember what you are giving how much credit for than if you graded all of exam one, then all of exam two, then ... . 

It also eliminates an important bias in grading—the tendency to form an opinion of a student based on his answer to one question and then let that opinion distort the grade you give him on another. As my wife pointed out when I was discussing this recently with her, that is a problem discussed by Daniel Kahneman in his (very good) Thinking Fast and Slow. He actually did an experiment, and found that grading question by question instead of exam by exam resulted in the performance of students coming out less consistent—made it less likely that the student who got a good grade on question one also got a good grade on question two. He concluded that he had a strong bias towards imposing consistency on his grading, even when it wasn't there.

Unfortunately, none of this is present in Computest. It would be nice if someone involved in producing the program happened to read this post and did something about it, but not very likely.

But not all of it has to be done by them. It should be possible to write a program that would take the output of Computest, the files representing what each student wrote on the exam, and reorganize it so that the professor doing the grading could grade by question instead of by exam. I cannot think of any comparable way of fixing things at the other end, of making it easier to take the exam, but perhaps one of my readers can.

What started me thinking about this was the recent experience of getting exams that had been taken using Computest on a flash drive from my school, grading them on my computer and returning them by giving the flash drive with graded (and commented) exams on it to the students to pass among themselves, with each copying his exam. It would have been  easier if I could have emailed the exams to them—for one thing, that would have given them a chance to look at them before class and see if they had questions. But Computest requires the student's ID number, not his grade, and under my school's blind grading system that means that I don't know the name and so cannot email the exam back. I can imagine a number of ways of fixing that problem, but do not know if any will prove workable.


Roman said...

Certainly, the software could do the emailing job for you. Especially if it's a web-based app. It wouldn't show students emails to you, but when you finish grading, it would email all students their grades without you having to do anything else.

I think the biggest problem why innovation in education is so slow and why developers don't seem to have enough incentive to invent anything better is because education is a very government-like industry in which processes are slow to change and it is very hard to pitch a new product.

Joe said...

Educational ICT systems have always been terrible from my perspective. My university in Norwich, England has only internet explorer, and despite having modern machines are slowed down considerably by many unnecessary processes in the taskbar.

I'm on a year abroad in Calgary now and I can see obvious improvements. Computers contain firefox, with a single antivirus program running in the taskbar. I can load all the tabs I like without crashing, and yet this is done with the same level of hardware.

Many government workers seem to be capable of "professional-looking" things, such as regularly upgrading to the next version of MSoffice, while being blind to actual and more important flaws. I tend to regard them as pseudo-experts, people who have the formal education and experience to impress a computer-illiterate management, without having the innovation to create a smooth system.

lelnet said...

I think that you may be making an erroneous assumption, to wit, that the software decision-makers (when taken as a group) value the elimination of bias as much as you do.

Their observed behavior, in this matter and others, tends to indicate that they do not.

Don't discount the utility of familiarity of workflow process, either. A teacher strongly accustomed to grading entire exams one at a time, instead of grading all responses to one question before commencing to grade all responses to another may be highly resistant to change. And from their perspective, this resistance might even be rational. Even if not, it will not be lightly surmounted.

Ori Pomerantz said...

Matt, this could be a user parameter. Let the teacher choose whether to do test by test or question by question.

I agree that the elimination of bias does not seem to be a high priority in a lot of academic settings. Then again, I believe the future is more things like edx and udacity, and less credential-based education.

cpwegener said...

Dear Professor Friedman--
Good article.
This feedback should be directed at the company that provides the test software. This sort of suggestions are great for software companies since I doubt it would be hard to implement and the programmers would never think of it on their own. (Most software companies, all protestations to the contrary, usually ship software that the programmers design.)
I would think the central server could sort out the emailing from student ID. Grading the test in question order with randomized answers would also be relatively simple given the input file.

Please email me at chris at wegenerconsulting dot com my google account is just because I have an Android phone

Anonymous said...

I would be interested if anybody knows of any open source test-taking software? If so you could likely get people to contribute to that and add the features you think are good.

On the so-called "bias" - I don't know, I used to grade engineering homework as a TA and I explicitly used what you're calling bias to help grade papers more quickly. For example if a student messed up a graph then a quick glance is enough to tell me if he messed up the rest also. If a student makes a certain kind of mistake over and over, that too is helpful. Since I had hundreds of papers to grade, and it was difficult to check every problem, I was forced to just check a subset of problems on each paper. In those circumstances heuristic and skimming are necessary to make up for that.