After answering the exam questions you know the answers to you still have time left. You spend the rest of your time answering questions you
do not know the answers to, in the hope that something you write will
fool the professor grading the exam into thinking you know the answer,
at least in part, expressed it unclearly and deserve partial
credit. Doing this is rational behavior on your part, but the result is to waste your time writing and my time reading. It also adds additional noise to the signal that exams generate, since there is a
risk that I will either be fooled into giving you credit you do not
deserve or interpret some other student's poorly written answer as
entirely bogus when it is not.
I have a solution to this problem, an economic solution, although I like to claim that it was inspired by the story of Socrates and the Delphic Oracle.
Well, one day [Chaerephon] went to Delphi, and there he had the impudence to put this question -- do not jeer, gentlemen, at what I am going to say -- he asked, "Is anyone wiser than Socrates?" And the Pythian priestess answered, "No one." Well, I was fully aware that I knew absolutely nothing. So what could the god mean? for gods cannot tell lies. For some time I was frankly puzzled to get at his meaning; but at last I embarked on my quest. I went to a man with a high reputation for wisdom -- I would rather not mention his name; he was one of the politicians -- and after some talk together it began to dawn on me that, wise as everyone thought him and wise as he thought himself, he was not really wise at all. I tried to point this out to him, but then he turned nasty, and so did others who were listening; so I went away, but with this reflection that anyhow I was wiser than this man; for, though in all probability neither of us knows anything, he thought he did when he did not, whereas I neither knew anything nor imagined I did."
On my exams, knowing what you do not know is worth something—twenty
percent to be precise. That is what you get on a question for not doing it. So if you
suspect that the best bogus answer you can come up with will be worth
less than twenty percent, you are better off leaving the question blank
or writing "I do not know," going home early, saving both of us some time and saving me the hassle
of trying to figure out which answers are or are not entirely bogus.
I posted on this and some related matters five years ago but thought it worth posting on it again.
I posted on this and some related matters five years ago but thought it worth posting on it again.
13 comments:
Heh. I am currently wading through some really impressive attempts to write a short answer about a passage (I am teaching an intro to Lit course) that explains its relevance without (I suspect) the student having any idea where the passage is from. Some are so blandly generic that it takes me a few minutes to realize they didn't say anything. This is a tempting solution ....
Excellent! I remember reading your thoughts on this some time ago and have adopted this practice.
Isn't this how the SAT works? You get a worse score by answering wrong than for not answering. The rationale there is a bit different: If you get a wrong answer, then you must have guessed (incorrectly). If you're the kind of person who guesses, then there's another question where you guessed (correctly). So we'll take that score away too. (E.g., with 5-answer multiple choice questions, you guessed at 5 questions, so the four you got wrong should subtract 1/4 of a point each.)
Lawrence:
I believe that's how the SAT works. I can't use a mechanical formula to give guesses a zero expected value because I am not giving a multiple choice exam. And telling students that they get points for knowing they don't know the answer is likely to be more popular than telling them that they lose points for giving a bad answer, even though they are mathematically equivalent. Also easier for me to do.
David, how often do students actually take advantage of the 20% option?
Eric:
Pretty often.
I give exams that tend to have fairly low average scores, so it isn't as if giving up 80% of the points on a question is suicidal. For the midterm I just graded, the mean score was 56/100.
David,
Your producing exams that generate such a low mean score is fascinating to me. Would you share your rationale for that? Do at least half your students fail your class or are there other components of the final grade that offset such low exam grades?
I ask not as an educator but as someone who spent a relatively long time as a student, the final few years taking classes for which the final grade was solely based on midterm and final exams. So even just thinking about such low scores is stressful.
I imagine that it depends what you're trying to test for and measure. If you're mostly interested in testing for a baseline of knowledge that you want even an average or low-average student to have, you probably end up with fairly high grades with above-average students clustered near the top of the scale.
On the other hand, if you're at least equally interested in drawing a distinction between OK students, good students, and exceptional ones you'll have a harder test and a lower average score.
William:
I don't give particularly low grades. Are you assuming that there is some fixed relation between numerical score on the exam and letter grade?
For that midterm, I told that students that, roughly speaking, 80-100 was an A, 60-80 a B.
Insofar as there is any theory to the exam, it's that I am trying to get information about how good each student is and what things students do or don't understand, information for both me and the student. A question that everyone can answer generates no information.
The course has no homework, so the midterm is the first point at which either I or the student is getting significant feedback. One problem in teaching economics--the course is on the application of economics to law, for law students--is that it is possible to think you understand something when you don't, when you are substituting some vague things you think you know already for much more precise and quite different things you are supposed to learn.
> One problem in teaching economics--the course is on the application of economics to law, for law students--is that it is possible to think you understand something when you don't, when you are substituting some vague things you think you know already for much more precise and quite different things you are supposed to learn.
This seems like a good description of the entire field of economics.
What a coincidence!
A course I've just finished takes a similar approach:
you get 20% of the average score of the other questions you answered.
"And telling students that they get points for knowing they don't know the answer is likely to be more popular than telling them that they lose points for giving a bad answer"
Ironic, given that the students are supposed to be learning economics....
In case anyone is interested, there's an article calculating the (as they argue) single best way of scoring multiple choice tests at http://cs.au.dk/~mis/multiple.ps
It gives an expected score of 0 if you're just guessing (or checking none/all the answers), it obeys some obvious monotonicity criteria (more points for checking fewer answers and for having the correct answer among the checked ones), and it's invariant when you combine two questions into one (cartesian-product the answer sets).
Amusingly, the article (like your post) refers to Socrates.
Post a Comment