I commonly write about the topic of error in assigning grades in a classroom setting, but it always fills me with a certain measure of anxiety.
Assigned Grade = True Grade + Error
This is a fundamental relationship that we have to deal with when trying to measure anything interesting. Yet I often wonder how students reading this blog might feel to be so frequently reminded that the grades they receive in their classes always contain some error. Though error never completely avoidable, it is easy to understand how someone might feel disaffected or angry to learn that the measures which hold sway over their future career prospects contain some degree of error. The degree of error varies from class to class and student to student, but it’s easy to imagine that in almost every class there are some students whose grades are in the grey area between, say, a B and a C, for whom even a small amount of error could be enough to produced an assigned letter grade that is different from what the ‘true’ grade would be if the error had not existed.
How important is the difference between a B and a C? In the case of Megan Thode, a student at Lehigh University, the difference amounted to a $1.3 million lawsuit.
To quote the NY Daily News:
Getting a grade you don’t deserve in school is worth about $1 million in damages, according to a lawsuit filed in Pennsylvania.
A former student at Lehigh University is so unhappy with the C+ she received in a course in 2009 has decided to sue the school for $1.3 million, claiming the unfair grade has ruined her future earning potential.
In this case the C+ grade was enough to disqualify Ms. Thode from continuing in her graduate program. Her lawsuit claims that the grade she received was inaccurate at least in part because bias the instructor carried towards her due to certain political statements she had made and complaints she made about taking part in an internship as part of the class. The university denies any bias.
For the sake of argument, let us assume that the instructor did not make a conscious decision to alter her grade, but that annoyance at Ms. Thode’s behavior subconsciously contributed to the error present in her assigned grade. It would not outrage me to learn that this was true. I personally do not consider myself immune to small biases based on student interactions – a point that I try to make clear to all my students when discussing my professional behavior standards and one of the reasons I employ blind grading for most assignments. This hypothetical situation provokes a few interesting questions:
- How much error is enough to warrant a lawsuit?
- If we hold instructors legally liable for grade error, does it matter whether the largest source of the error is unconscious bias, sampling, grader variation, or any of the many other factors that can contribute to error?
- Is it even possible to determine how much of an assigned grade is error or what caused the error?
I’ll pass on trying to answer the first two questions as they are mostly philosophical. The question of determining the magnitude and cause of error is a practical question. In theory, the answer to the question is ‘yes’, but with several caveats so large as to make any attempt clearly impractical.
Classical testing theory posits that while the sampling error in each assessment is random, the randomness is not uniform, but rather forms a normal distribution centered on zero. From a statistical standpoint, all that you would need to do to reduce the error is to continue sampling, provided – and here comes the big catch – that you could ensure that the thing you are sampling (i.e. the student’s knowledge) has not changed at all, something that would require access to a time machine or alternate dimensions. As for rooting out bias-driven error, it probably isn’t normally distributed, so you might be able to discover it if you were able to fit multiple independent graders into your time machine so you could re-test each student in the class multiple times and then try to correlate the scores between graders.
In other words, short of some kind of smoking gun, it is not practical to sue based on standard assessment error. Frankly I’m glad, since the thought of the chilling effect that would have on grade assignments is enough to make me shiver.