You use AI to speed up feedback on a stack of essays. It's quick and it reads well, so you send it out.

What you don't see is that it's given one student notes on their argument and another notes on their spelling, for the same quality of work. It got there on assumptions drawn from the student's name or a label it picked up from your systems.

A Stanford study gave AI the same essays with a different student description each time. The feedback changed, and it changed along familiar stereotypes.

If you're looking at AI tools that write feedback on student work, put this paper on your reading list.

Researchers at Stanford tested what happens when you tell a large language model who the student is. Same essay, same task. Only the description of the student changed.

And the feedback was different, sometimes very different.

What the study did

The team took 600 persuasive essays written by US eighth graders (Year 9 in England). They ran each one through four models: GPT-4o, GPT-3.5-turbo, Llama 3.3 70B and Llama 3.1 8B.

Each model was asked to act as an English teacher giving revision feedback. Sometimes the prompt said nothing about the student. Sometimes it described them by race, gender, English language learner status, learning disability, achievement level or motivation.

Then the researchers counted which words each version of the feedback leaned on.

What changed in the feedback

Describe the student as an English language learner and the feedback swung towards grammar and spelling. That language rose 78% on the baseline, and comments on the actual argument thinned out.

Students described as low achieving got short, corrective feedback on mechanics. Describe them as high achieving and that corrective language dropped by 40%, replaced with praise and stretch.

Students with a learning disability were steered towards shorter, simpler writing.

The race findings are the starkest. Language the researchers linked to Black students rose 180% when the essay was labelled as written by a Black student. That feedback leaned on personal experience and collective identity. Hispanic and Asian students got more language correction. White students got pushed on argument structure and evidence.

Girls got warmer, more emotional language. Boys got more task-focused critique.

The authors point out that these patterns mirror biases already documented in human teachers. The models appear to have absorbed habits we already know exist in classrooms.

What about names?

The researchers also ran a smaller test using first names alone, with no description of the student. The effect there was weaker and less consistent.

But it wasn't zero. Some identity-linked language still crept in based on the name, which tells you a model doesn't need a label to make an assumption.

What this means for schools and colleges

"Personalised feedback" is a selling point for a lot of these tools. In practice, personalisation means the tool knows something about the learner. The paper notes that can arrive through your learning platform, or be inferred from a name or the writing itself.

So before you switch one on, ask the supplier what student data the tool sees and whether they've tested for this kind of bias.

And don't assume a newer, bigger model fixes it. The authors say plainly that it won't.

This is unconscious bias at scale. A teacher with a blind spot affects one class. A tool with the same blind spot repeats it on every essay it's been used on, and nobody sees the pattern unless they go looking for it.

So ethical AI use means naming the problem. If your AI policy doesn't mention racism and discrimination as risks, add them. Schools and colleges already have duties under the Equality Act 2010, and those duties still apply when a tool wrote the feedback.

As always, the best defence is to remain the human in the loop and know your learners. An AI draft of feedback can save time. Checking it for these patterns before it reaches a student is still your job.

Would you spot it if your feedback tool was doing this?

Sources

Tan, M., Phalen, L. and Demszky, D. (2026). Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback. Stanford University. LAK 2026, Bergen.