What automated grading can and cannot fairly judge

A rubric can make some parts of feedback inspectable. It cannot make every judgment automatic.

Stanford Teaching Commons recommends aligning learning objectives, assessments, and criteria, discussing rubrics with students, and reviewing chatbot-generated feedback for accuracy and relevance. UNICEF’s guidance also recommends human oversight for critical functions and some assessments, including high-stakes summative assessment. These are educational recommendations, not proof that automated grading is fair, accurate, or appropriate in every setting.

Consider a fully fictional writing task. The objective is to write an explanation of exactly 120 words about how a fictional town’s recycling plan works. The rubric has three criteria, each worth one point: the response explicitly states that glass goes to the blue bin, paper goes to the white bin, and collection occurs on Friday. The student’s response says: “Glass goes to the blue bin. Paper goes to the white bin. Collection is Friday.”

An invented automated feedback draft could check visible alignment with the rubric: it identifies all three required facts, then reports that the response is shorter than the exact 120-word target. That is a mechanical comparison. The teacher can inspect it against the response and rubric.

The same system should not silently decide every question around the assignment. It cannot infer from this fictional response whether the student understood the distinction between the materials, copied the sentence, or received an accommodation that changes how the work should be interpreted. Those judgments require the assignment context and the teacher’s knowledge of the assessment.

The teacher also has to inspect the rubric itself. If the learning objective is explaining why the recycling plan works, a checklist of bins and day may be too narrow. An automated comment can be accurate relative to that checklist while the checklist measures the wrong thing. Accuracy and relevance are separate checks.

My practical division is three-part. Let automation compare visible text with explicit criteria. Let the teacher verify the comment against the student’s actual response and the intended learning objective. Let the student see the criteria and have a way to question a comment. If the task is high stakes, the case for human oversight becomes stronger under the guidance cited here.

A grade is a judgment about work under stated criteria. An automated comment can help expose whether certain criteria appear in the text. It cannot, by itself, establish that the criteria were sufficient, that the interpretation fits the student’s context, or that the final judgment is fair.

The useful question is therefore: what exactly is being judged, and who is accountable for deciding that it is the right thing to judge?