Monday, November 26, 2012

The Use of Old Exams

University libraries often keep a file of old exams, at least for those courses whose professors approve of the idea, and make them available to students. As best I can tell, there are two reasons they do so. One is to help students study for  exams they are going to take. The other is to prevent students who have access to old exams from other sources, a friend who took the course the year before or a fraternity that keeps a file of old exams provided by its members, from having an advantage over students who lack such access. My own practice is to cut out the middleman by webbing some of my old exams and linking to them on the class web page.

I have, however, some reservations about the practice, having to do with how the old exams are used by students. The way I want them to use the exams is as a way of checking on how well they know the material, so that if they think they understand part of it and don't they will discover the problem before, not after, taking the final. My usual suggestion is that, after studying, a student should take one of the webbed exams and use my answers, if they are there, to check his. If the answers are not there, he can at least go back to the book to see whether what he wrote fits what it said.

What I do not want the student to do, and am concerned that many students may try to do, is memorize the answers to all the question on past exams on the theory that those are the questions that will appear on the next exam. One problem with that is that you can memorize an answer without understanding it. Another is that the exam questions, even from multiple exams, cover only a fraction of what the students are supposed to have learned; an exam is a sample of the course, not a summary. If I limited my exams to questions from the old exams that I have webbed, a student might be able to get a reasonable grade by memorizing answers to those questions, but the grade would be poor evidence of how much of the course he understood. Memorizing answers is analogous to the practice of going through a textbook using a highlighter to mark the five or ten percent that you believe you actually are supposed to learn—or at least will be tested on.

If I try to avoid including in the current exam questions that were in the webbed past exams—which is mostly what I do—a student who studies by memorizing answers will not only waste his time in a long run sense but in a short run sense as well, since not only will he not have learned the subject and be unlikely to remember much of it a year or two later, he will not even get the good grade his effort was intended to produce.

My problem as a teacher is how to get the benefit of making it possible for the student to use the exams in the way I want him to without making it too likely that he will use them in the way I do not want him to. I do not have a really satisfactory solution. I tell my students how I want them to use the old exams, but students, reasonably enough, may suspect that my objectives are not identical to theirs, hence that advice it is in my interest to give them may not be advice it is in their interest to follow. I also warn the students that I try to avoid putting questions from the webbed exams on the current one, which may be more effective, providing they are paying attention, believe me, and remember.

One element of the problem is the question of whether to web answers as well as questions. One of the problems in economics, in my experience, is that because it deals with features of the world that students are familiar with and uses ordinary language, often with specialized meanings, a student may go through a course thinking he understands everything but the fine points and end up having learned almost nothing. Having done so he might answer all the questions on an old exam to his satisfaction but not to mine. Providing answers makes it easier for a student to tell whether he actually understands the subject—by how well his answer fits mine.

The disadvantage is that students may take the opportunity to memorize the answers instead of learning the course material.
At some level, my response to all such issues is that it is my  job to make it possible for my students to learn, theirs to make it happen. If a student chooses to ignore my advice and devote his efforts to memorizing answers in order to get a good grade on the exam, rather than learning ideas in order to understand what the course teaches, that is his responsibility, not mine. Along similar lines, I make no attempt to enforce compulsory attendance. But I would still prefer, so far as I can manage, to teach the course in a way that will make it more likely that students end up understanding the ideas it covers.

Having discussed at some length one issue associated with giving exams—it is, of course, that time of year—I will take the opportunity to mention two others, starting with a policy I adopted years ago designed to make taking exams a little pleasanter for students, grading them a little pleasanter for me, and the resulting grades a slightly better measure of what each student knows.

Imagine that you are a student taking an exam, and after answering all of the questions you know the answers to you still have some time left. It is tempting to spend the rest of the time answering the questions you do not know the answers to, in the hope that something you write will fool the professor grading the exam into thinking you know the answer, at least in part, expressed it unclearly, and deserve at least partial credit. Doing this wastes your time writing, my time reading, and adds some additional noise to the signal that exams generate, since there is a risk that I will either be fooled into giving you credit you do not deserve, or interpret some other student's poorly written answer as entirely bogus when it is not.

My solution to this problem was inspired by Socrates' explanation of why he was, as the oracle told him, the wisest man in Athens. He was initially dubious, since he didn't know anything. But, after extended conversation with his fellow citizens, he concluded that they didn't know anything either—but thought they did.

On my exams, knowing that you do not know something is worth twenty percent. That is what you get on a question for not doing it. So if you suspect that the best bogus answer you can come up with will be worth less than twenty percent, you are better off leaving the question blank or writing "I do not know," going home early, and saving me the hassle of trying to figure out which answers are or are not entirely bogus.

My other policy, adopted several years ago, is to give short exams, exams which I expect most students to finish before their time runs out. My original reason for doing so was my dissatisfaction with the common practice of giving students who can persuade the relevant university officials that they have some invisible handicap, some sort of learning disability, extra time on exams. While some of those students may suffer from a real problem, I suspect that in many cases all that is special about their situation is having parents willing to pay a professional to produce the needed diagnosis.

I did not like being a party to what I regarded as legalized cheating.  I had no way of preventing it, but I did have a way of making it ineffective. If everyone can finish the exam before time runs out, having an extra hour is no longer an advantage.

That was my original reason for trying (not always successfully) to write short exams. After I had been doing it for a while, I concluded that it was a good idea on its own merits. Being able to do things fast is sometimes useful, but in most contexts getting the right answer is more important than getting it quickly. An exam that most students find it hard to complete rewards speed by more than I think it should be rewarded.
It occurs to me that there is one more policy of mine with regard to exams at least worth mentioning. I only write the exam after the last class. That way I do not have to worry, when students are asking questions in the final review class, that I might be giving away the answer to an exam question, unduly advantaging those paying attention at that moment and reducing the ability of the exam to function as a random sample of the student's knowledge.

And, for a last comment ...  . I like to say that being a professor is better than working for a living, except when grading exams. One reason is that grading exams is a pain. Another is that it is when you find out that you have not done nearly as good a job of teaching as you thought you had.

13 comments:

Anonymous said...

Here's an idea that might help reduce the incentive to memorize answers: Include something intermediate between an answer and no answer. Often this could be a strong hint, or it could be a partial explanation with the last step or minor details left out. This helps students who genuinely tried to answer the question, and dissuades students looking to memorize since a tiny amount of effort would be required to produce the answer to each question.

I'm just speculating, but this approach might work best if the questions are presented in a structured format such that work in correctly answering one question is rewarded when trying to solve a later one. This way students will actually feel like they gain something from trying to figure out your old exam questions, as opposed to regarding it as a chore or a list to be memorized.

Anonymous said...

I am impressed that a professor has noticed "extra time" is often for bogus reasons.

An additional use of past papers is to help filter out what material is part of the course and what is just background interest. Some courses are not clear enough to do that naturally.

Another is that it is when you find out that you have not done nearly as good a job of teaching as you thought you had.

That isn't always obvious. There are courses which I only really understood a year or even two after the final exam - so the benefit I gained from good teaching was never recorded.

William H. Stoddard said...

It seems to me that what you take as the pathological effect of exams being made available may well be the actual purpose of doing so, as far as universities are concerned. Offering a look at past exams gives students the ability to focus their study effort on just the areas that are required to pass, without actually mastering the subject. That's bound to be popular with a lot of students. It lets a university administration gain popularity with students, and appear "responsive" to student desires, at what seems to them to be a low cost—even if it actually hinders learning.

jimbino said...

The best way to solve your exam problems, as well as many others, is to grant everyone a PhD, MD and JD at birth. Then classrooms would stop being polluted by students who didn't want to learn and those whose chief qualification is rich parents.

Just think how happy professors would be, teaching, like Socrates, only those who were desperate to improve themselves.

Of course, few profs measure up to Socrates; the others will always need classrooms filled with dumb students with rich parents.

Gary Y. said...

Crowd Source it!

Have an online 'discussion' after each lecture and a standard topic perhaps "What are the 'three' most important ideas presented in this lecture?"

Then have a round where the various similar suggestions are coalesced, then another round where the ideas presented are voted on and ranked. (Negative numbers could mean various degrees of "wrong!" )

This initial ranking could then be followed by discussion and re-ranking, elimination of lowest ranked suggestions, discussion, re-ranking until the top 'three' suggestions remain.

Depending upon the quality of the students and their participation, it could well happen that the students would learn more from active participation in the online discussion than they did during the lecture. :)

The instructor would certainly get immediate feedback on efficacy of his or her lecture and, if the students have collectively misunderstood something important, the matter can be addressed in the very next lecture.

It seems possible that students who regularly participate in the discussion could be excused from a final exam or, if records are kept of 'performance' in the discussion, that students could accumulate 'points' sufficient to excuse them or make the grade points achievable on the final redundant.

A similar procedure could allow students to propose test questions -- or even grade the answers.

G.

dWj said...

I'm still pretty new to making and giving exams, but have moved at this point to open books and open notes for all tests, in large part as an attempt to get them away from spending too much time trying to memorize material rather than understand it. This is related to one of the reasons you (and I) try to keep them short, too; the ability to do problems quickly on a desert island may occasionally be handy, but ultimately the ability to do things more deliberately and with access to the usual array of materials will be of more importance (though I do think this purpose can be advanced at times by inducing students to memorize material and practice doing problems quickly).

Phil Birnbaum said...

(I commented earlier, and it seemed like my comment was accepted. Now it's gone. Was it held for moderation?)

Rebecca Friedman said...

You hardly have to have anything that fancy for an online discussion to produce productive results - if you can get people to participate in it.

One of my favorite professors had for two of his courses a discussion forum. Students were encouraged to post questions frequently, and then to respond to each others' questions. This resulted in the students explaining the material to each other, which at least for me was very productive - I felt much more motivated to double-check my sources and reasons for reading things the way I did when there was another student going "Where did you get -that-?"

... Of course, that was a literature course. I don't know how much the same ideas would apply to economics.

Tibor said...
This comment has been removed by the author.
Tibor said...

An interesting article! Some notes from me (I don't know how useful as economy and mathematics differ considerably):

1) Why not have oral exams instead? It depends on the amout of students that take your class, but if possible, it gives a much better info about who actually understands the topic. You can give out a list of broad topics before the first exam term and then have each student draw a card from a box or generate a pseudorandom number each associated with a topic. Then talk about that with the student, have him draw another one (more topics - reduction of the chance of the student being just "lucky" with the first draw) and repeat the process.

2) If the possibility of examining each student orally is out of hand due to too many students, have a combined written/oral exam. A written exam that is aimed to cover the width of student's knowledge, with a lot of rather easy questions from various part of the lecture and then an oral exam for those who pass that on one or two topics as described in 1).

2b) An interesting method used by one of the professors at my school - you draw just one card, but if you are not satisfied you can draw again - which reduces your final grade by one step. You can repeat drawing if not satisfied until the resulting grade would be a failure for sure anyway (which means you can change the card twice at our school). That again fixes the problem of luck a bit...or bad luck in this case. A student that knows all but the last topic and he picks that, can still pass quite well.

3)In parts of the exam where there is not an explicit demand for definitions or things that can be simply copied from the book, let students have the book(s) with them at the exam.

The following file is one of my exams (it is in english) in the last semester (the teacher having the same policy as you do...he provides the answers in a separate file):
http://staff.utia.cas.cz/swart/tentamen12.pdf

I was allowed to use all the material for the class at the exam. It helped me to check some minor details in definitions and such, but had I not understand the subject it would have been of no use.

If you formulate the questions in the way that for a student who doesn't understand the topic the book is of very little use, it is a good way to reduce the "information distortion". And it also eliminates the option of cheating. You can already have any book, so nothing can help you. It is definitely possible to make such exam questions in mathematics (or physics), but I am not sure if it is in economy.

4) I like your policy of more-time-than-needed exams. I often find myself doing a far better job at the exam when I know I don't have to worry about the time (which is fortunatelly all of them now at the master's level when I am often the only student at the exam term). I even think faster that way. And I believe that at the end of the day it is about who has the knowledge and skill and can figure the problems out and not about who can think the fastest (or is most resilient to stress from time pressure).

RKN said...

I liked the approach of my biochem professor years ago. His exams had a list of questions every student must answer, for ~50% of the total points (iirc), and a list of other questions the students could choose from to achieve the remaining 50%.

If students had extra time they could provide answers to more questions than necessary in the optional list, increasing the possibility of getting 100% of the total points, or even more.

In that class we were sometimes provided exams (both Q&A) from prior years, but it was made crystal clear to us not to expect any reuse of those questions on the current year's exam.

Jonathan said...

I got my first driving licence in Italy. At least at that time, there was a theoretical test of 20 multiple-choice questions, all taken from a published book of 2000 such questions. To prepare for the test, I bought and went through the book. Most of the questions were easy, so I concentrated on the small proportion of more difficult questions and memorized the answers. I did very well in the test; better, I think, than most native Italians, who were less methodical about it. This is just a little anecdote that comes to mind in this context, make of it what you will.

Anonymous said...

I'm also a prof (undergrads at a SLAC), and also favor short tests, but came to them in a different way. Even among students who do not have fine motor issues, handwriting and typing speed vary hugely -- and, critically, not in ways that correlate with knowledge of and insight into the class material.