For most of the year, being a professor is better than working for a living. The exception is when I am grading exams. Part of the reason is that I find out how much worse a job I have done than I thought I had—how many students don't understand things I thought I had taught. Part is the time and trouble of evaluating their answers.
And part is trying to read their handwriting. That is the part that can be solved.
Nowadays, most students have computers and are used to using them. Twenty-some years ago I came up with an idea for software to let students take exams on computers. The project, which never got very far, was going to be called Electric Bluebook. Many years later, someone else did it; the law school where I do most of my teaching uses their software to let students take exams on their own laptops
When I was designing Electric Bluebook I was mostly concerned with ways in which the computer could make taking and grading exams easier. My version would have let students keep track of which questions were unanaswered, which finished, which they wanted to go over again if there was time. At the professor's end, the software could show the professor all answers to question 1 in random order, thus eliminating the problem of shuffling bluebooks or letting a bad answer on one question bias the grading of another.
The software that was actually written had a rather different set of priorities. It was designed to solve one and only one problem: cheating. The programmers faced, and I gather solved, the difficult problem of how to let a student use their software while locking him out of everything else on his computer—which might include class notes, answers to old exams, even a downloaded textbook. I liked my design better, but I understand the depressing reasons for theirs.
Currently I am grading exams from a course I volunteered to teach at a different and poorer school, one which does not have such software, with the result that all exams are handwritten. It occurred to me that, as the price of computers falls, one solution would be for the school to have a bunch of cheap laptops dedicated to the sole purpose of taking exams on, wiped clean of everything else after each use.
Then it occurred to me that there is a much cheaper solution, a readily available source of free computers for the purpose.
Match up two classes that take their exams at the same time. Tell the students that if they bring a laptop, carefully labeled with their name, they will probably be able to take their exam on a laptop--but not theirs. Advise them to password protect anything on their laptop they want to keep private. Before the exam begins, randomly trade laptops between the two classes. Put the exam for each class on all laptops used in that class. When the exam is over, transfer the files representing the completed exams to the professor's computer and return all laptops to their owners.
One can imagine a variety of practical problems, but I think most of them have straightforward solutions. Students no longer have to scribble, professors are saved from eyestrain, and the only cost is a little time spent shuffling laptops.
Comments?
And part is trying to read their handwriting. That is the part that can be solved.
Nowadays, most students have computers and are used to using them. Twenty-some years ago I came up with an idea for software to let students take exams on computers. The project, which never got very far, was going to be called Electric Bluebook. Many years later, someone else did it; the law school where I do most of my teaching uses their software to let students take exams on their own laptops
When I was designing Electric Bluebook I was mostly concerned with ways in which the computer could make taking and grading exams easier. My version would have let students keep track of which questions were unanaswered, which finished, which they wanted to go over again if there was time. At the professor's end, the software could show the professor all answers to question 1 in random order, thus eliminating the problem of shuffling bluebooks or letting a bad answer on one question bias the grading of another.
The software that was actually written had a rather different set of priorities. It was designed to solve one and only one problem: cheating. The programmers faced, and I gather solved, the difficult problem of how to let a student use their software while locking him out of everything else on his computer—which might include class notes, answers to old exams, even a downloaded textbook. I liked my design better, but I understand the depressing reasons for theirs.
Currently I am grading exams from a course I volunteered to teach at a different and poorer school, one which does not have such software, with the result that all exams are handwritten. It occurred to me that, as the price of computers falls, one solution would be for the school to have a bunch of cheap laptops dedicated to the sole purpose of taking exams on, wiped clean of everything else after each use.
Then it occurred to me that there is a much cheaper solution, a readily available source of free computers for the purpose.
Match up two classes that take their exams at the same time. Tell the students that if they bring a laptop, carefully labeled with their name, they will probably be able to take their exam on a laptop--but not theirs. Advise them to password protect anything on their laptop they want to keep private. Before the exam begins, randomly trade laptops between the two classes. Put the exam for each class on all laptops used in that class. When the exam is over, transfer the files representing the completed exams to the professor's computer and return all laptops to their owners.
One can imagine a variety of practical problems, but I think most of them have straightforward solutions. Students no longer have to scribble, professors are saved from eyestrain, and the only cost is a little time spent shuffling laptops.
Comments?
25 comments:
The proposal is actually very clever. There is one huge problem though: Technical difficulties. What do you do if one computer has a problem, or breaks down entirely?
Check out http://www.nomachine.com/
Students could log on and then you could lock their computer that they are on to that screen alone-no doubt it could be set to do it automatically. This effectively makes their computer a thin client for the remainder of the exam- they can only access what you allow them to get to on the server and they can't touch anything on their computer until they exit out- which will be after they are finished with the test.
David,
Do we then reduce the prof's pay to compensate for the costs of setting up the system and the reduced workload he will have?
If I understand correctly, the laptop shuffle is a way of preventing cheating. In particular, it tries to deny access to information previously stored on the laptop.
Well, what if the students store the valuable information on memory sticks instead? Most (all?) laptops come with USB ports nowadays, therefore accessing the information on the memory sticks won't be a big problem.
A.M.
Students could sabotage each other, wittingly or not, with subtly broken laptops.
One could use Damn Small Linux or T2 to create a bootable CD or USB drive, in such a way that it could not be rebooted without a password. This would provide a closed environment for hosting the exam-taking software. The students would bring their laptops and would have to boot from the CD/USB.
I would second what Joe mentioned, as my previous experience with Linux leads me to believe such a setup would require only a few hours to develop, and only a few more to deploy.
However, wouldn't it be more cost effective to give each classroom a set of AlphaSmart Neo's? You can't do anything except input text--that's all the device does. Then you could sync the answers to the exam to the professor's computer after class. An AlphaSmart Dana has wireless access, meaning the exam answers could be posted to a central repository. Since the Dana is based on Palm OS software, I'm fairly sure the big ol' Internet could be locked down.
This doesn't solve the problem of grading, which should be done by hand in the case of essay-style questions. Surely your years of experience make it practical to create a rubric and grade on a simplified scale, and then assigning letter grades according to whatever distribution you prefer.
Plus, who's better qualified to grade your exams than you? Jimbino's objection would be of no concern.
I'll third Joe's suggestion. The graphic installation on most linux distros is pretty low-key as far as what has to be loaded so there could be a low-fi loading of Xorg (the basic, no-programs visual interface software for linux machines) and the only application that could run would be the electric bluebook.
Like Joe said, there are all sort of bootable OS's like knoppix which detect the actual hardware of the computer. This means it's just a matter of distributing the CD's.
As far as the laptop problem goes, I wouldn't encourage trading of laptops for liability reasons. Even if the school isn't blamed in some abstract way for a student breaking another's laptop, it still seems like too close of an association. Still you could maybe offer some sort of fudge-factor extra credit or maybe the prospect of not having to physically turn in the exam might be enough to get at the very least _more_ people to use it.
And from what I gather, if only three people do participate, at least that's three exams you don't have to strain your eyes on.
Hi,
I'm a Computer Engineering Student and i think your idea is very interisting, but i think there is a saolution that could be a bit cheaper.
Now a days, there are free code operational systems. It could be created an use dedicated one to be booted in the computers of your students, where would run your examination software... the idea of the dedicated operational system (that is a bit complex to be programed.. but, who cares?)is that you could disable the support for every thing you want (except your software)...
do you understand my idea?
well... i think it's a bit dificult to make this OpSys... and the time spent to boot the machines is unknown to me... but this idea may be used in a lot of institutions.. certainly is a great improvement for the education method (and nobody is going to touch my laptop!)
[]'s
I do not trust others with my laptop,
nor am I willing to run foreign software on my laptop. And I think everybody ought to be as careful.
If given my own laptop and knowledge of the protection mechanism used, I'm fairly confident that I could come up with a way to circumvent it.
I even managed to program fake "are you sure yes/no?" + "memory cleared!" screens into my graphical calculator when I learned that they would have to be reset prior to taking the exams (just for fun, not to cheat).
I think the best solution is for the school to provide a thin client. These are cheap enough.
I am reminded of a method of teaching that an engineering professor of mine used. He assigned an enormous amount of homework from the book in the syllabus at the start of the year such that previously assigned homework was due every class. Then he flipped a coin at the start of each class and asked a random student to call it. If the student was right, the professor had us hand in the homework and he graded it.
The homework was worth 50% of the overall grade for the course, so it was unsafe to ignore it, and the professor slashed his workload in grading the homework by half. I thought it very clever.
It also seemed very effective at getting students to do their homework consistantly.
His name is Robert Lipp at University of New Orleans, I think he is retired by now.
This is similar to, but obviously not identical to, the "I cut, you choose" algorithm.
There was also an SF story 30+ years back (Niven? someone else?) about First Contact: a human ship meets an alien ship in System X. Neither one wants to go home ... if they go home, the aliens can watch them using sensors, and learn where the other race's home system is.
The solution is to destroy all the sensors on your ship, but leave the engines, life support, and navigation, and then swap ships.
anonymous:
The liveCD is pretty innocuous, it doesn't store anything on your computer, won't even access the hard drive. plus it would be coming from the school so we can assume it will be safe, otherwise the school _would_ be liable for damages.
Also, if what I suggested was used (Where it's basically just Xorg running with a GTK application taking up the whole screen) there will be no "protections" for them to circumvent, as the "protection" would be that the only interface to the computer would be through the electric bluebook (as literally nothing else would be running, no command line, no windows key+d for the desktop that isn't there, no start>run, three finger salute restarts the computer, etc) so it would be hard to imagine such a thing would have some sort of security problem.
I like the idea of forcing them all to use typewriters.
But no, I'm not being serious.
More seriously, it may be worth pointing out that in real life you're at liberty to refer to any sources of information you like. So why shouldn't you be able to do so in exams?
It seems to me that exams should test understanding, not memory. I'd prefer to see students allowed to consult whatever they like during exams, as long as they do so within the time limit of the exam.
This would make the exams more difficult to design, but it would provide a better approximation to real-life situations that students will later face.
Hi David,
I was diagnosed with Rhematoid Arthritis 9 months ago and have availed myself of all sorts of assistance to make writing exams easier. At both UVic and Camosun college there are special computers set up for people to take exams if they need to type their answers rather than write.
I am always so paranoid about breaking the rules when it comes to exams that when I have used these special computers for the exams I have never even attempted to look and see what info is available on the harddrive, OR try to connect to the internet.
They have set these computers up so that when you walk in, they are open to a word processor and after you are done you simply print out your document and that's the end of it. Nothing gets saved on the hard drive aside from the automatic document save.
Also - when you are writing these exams you are in a room by yourself where you are entirely visible at all times to the exam supervisor.
At both UVic and Camosun they have about 6 of these computers set up for exam writing for students with disabilities.
I know that both these schools also have huge computer labs which I think could also host the exam process.
I am thinking that with adequate supervision while the people are taking the exam (2-3 people patroling) the cheating problem would be no more problematic than during a regular exam room experience.
At the University of Michigan Law School they have already instituted such a program. Nearly all students have laptops and as such they take all exams on them. If you don't have a laptop, you can rent one from the University for cheap to take an exam. If your laptop crashes in the middle of an exam (which almost never happens) or if you simply prefer the old-fashioned way, they have extra Bluebooks for anyone who needs/wants to do the exam on paper.
From all I have heard the system has been a raging success. While there have been a few cases of people's computers crashing during exams, this can be prevented easily enough by using the rented laptops which are scanned and checked by the techie crew to make sure that such an occurrence won't happen.
"... the rented laptops which are scanned and checked by the techie crew to make sure that such an occurrence won't happen."
There speaks an optimist! All man-made devices can and will fail at least occasionally, however well checked. I suppose all the space shuttles were well checked, too.
But I'm not trying to make any point here except about slightly careless use of language.
I actually agree with jonathon, although I think that it would only be acceptable at higher level courses where there is less memorization required.
As long as I can use someone else's laptop to get connected by any means, I can access my data. I doubt this solution could robustly prevent cheating for this reason alone.
A solution which may be better than the "lock-down" approach is a logging software approach. A keylogger/process logger is installed on each laptop (each student keeps his own laptop). The professor opens a program which monitors the activity being sent from each of the loggers (live). The logging software sends a heartbeat to the professor's monitoring software every X seconds so that if the student attempts to shut down or disable the logging software, the professor will know in X seconds.
The professor can configure whatever settings in the monitoring software which he wants to be notified about, for example, opening any process other than the test-taking software, or opening any process except Microsoft Word, or Notepad, etc. etc. This way, if a student opens an internet browser, the professor will be alerted immediately.
There are two problems even with this, one probably fatal. The first is that many processes are started on timers and other random events so that the professor may be receiving a significant number of false alerts when RealMedia or other malware (yes, RealMedia is malware IMO) invokes itself out of nowhere. If the professor were to receive even one such false alert per exam, this would make cheating significantly easier since the professor would be much more likely to ignore or only cursorily investigate an unexpected process launch.
The other problem is fatal to this and the "lock-down" approach, and that is virtualization (Parallels, VMWare, Xen, etc.) With virtualization, the student can literally "switch virtual laptops" at the press of a key. Neither the lock-down approach nor the logging approach can prevent or detect this.
But, we can always go back to the fact that the vast majority of people are fair and play by the rules. I'm skeptical that any technological measure can reduce the real incidence of cheating which makes you wonder why we should bother with countermeasures. I would argue that preventing cheating is a similar problem (for the school) as media producers face with DRM. Every counter-measure simply raises the premium on defeating it.
The question that I think needs to be answered is if school A allows laptops without countermeasures, while school B only uses laptops with countermeasures, will the graduates of school A be of any lower quality (reducing school A's brand power)? It may be the case by using the time, money and effort expended on countermeasures for other uses, school A actually produces *higher* quality graduates.
Jonathan,
I only say this because the only time I've heard of a laptop crashing during an exam was when it was a student's and not one rented by the University. To be perfectly honest, the technology is developed enough at this point to ensure that laptops used solely for the purpose of taking exams will be crashproof.
Still, as any lawyer will tell you, hearsay evidence is among the least credible testimonies there are. I will try to find out if there has ever been a recorded case of a rented laptop crashing during an exam.
@Jonathan
"It seems to me that exams should test understanding, not memory. I'd prefer to see students allowed to consult whatever they like during exams, as long as they do so within the time limit of the exam."
Then how would the children of rich Senators graduate from the top-ranked schools? If they actually have to *understand* the topics instead of just memorizing the Pavlovian responses to the multiple-choice questions, then who graduates from the top-ranked schools is no longer just a matter of who has the richest and most powerful parents. ;)
(End of envious populist rant)
I'm one of the SJSU students with poor penmanship. Sorry.
Any chance we can get the answers to the final posted?
One possible solution to the virtualization problem on student-owned laptops would be to have a TA boot up each computer from a CD (more likely to be enabled by default than flash- or net-boot) into an environment that only provides access to the exam software, then remove the CD. Networking could be set up automatically on most machines to save the results to a central repository, with automatic notification that a flash drive is needed if networking couldn't be set up for some reason. Until something like the Xen hypervisor is loaded into boot prom's, to which the student could escape and switch operating systems, this seems like a pretty safe solution.
In the SJSU Econ classrooms, you would probably have to set up a generator outside the classroom to power the laptops that start to die during the exam.
Do as my colleagues do--give nothing but multiple choice exams.
That makes the only part of the our job that might reasonably be called 'work' go away.
Post a Comment