Automated IELTS-like essay scoring with GPT-oss-120b & Qwen3-32b, routed through a Human-in-the-Loop framework
We tested whether two large language models can score IELTS and TOEFL essays reliably, without systematic bias. If they can, automated scoring becomes a real option. If they cannot, any deployment carries hidden fairness risks.
GPT-oss-120b (Mixture-of-Experts) and Qwen3-32b (Dense) were each given the same 150 essay collection. GPT successfully completed scoring for 130 of them; Qwen for 128. Each completed essay received 5 independent scoring passes to measure self-consistency.
ICC tells you whether a model gives the same essay the same score when you run it multiple times. A high ICC means the model is self-consistent. A low ICC means the scores shift too much to be trusted.
GPT-oss-120b reaches an ICC of 0.943 on its 97 valid essays, putting it in the Excellent range. Qwen3-32b scores 0.844 across all 128 valid essays, firmly Good but short of Excellent.
A paired t-test on 95 essays scored by both models finds an average gap of 2.85 points between the two overall means (t = 7.43, p < 0.001). The Bland-Altman analysis puts the per-essay mean difference at 2.07 points, with individual essays varying anywhere from 7.38 points below to 3.25 points above.
The Bland-Altman plot shows this clearly. If both models scored the same way, the dots would scatter evenly around zero. Most fall below, pointing to GPT consistently applying a stricter standard.
These are the actual scores for all 95 essays both models evaluated, sorted from lowest to highest. Each point on the blue line is GPT's score for that essay. Each point on the amber line is Qwen's.
The two lines move together — both models rank essays in roughly the same order — but the amber line stays above the blue almost all the way across. That gap is the 2-point severity bias made visible.
Consistency measures how much a model's score changes when you run it again on the same essay. Two numbers capture this: Coefficient of Variation (CV) and score range across 5 runs.
GPT's mean CV is 0.047. Qwen's is 0.092, nearly twice as variable. GPT's average score range across runs is 2.4 points. Qwen's is 5.6 points.
Both models score essays differently depending on the prompt type. Essays that describe data, charts, or graphs receive the lowest scores from both models across the board.
The gap between models is widest for data report prompts at 3.17 points, and narrows for opinion tasks. Both models apply stricter criteria when the prompt expects students to interpret visual data that is absent from the text.
Human review routing flags any essay where the model's 5 runs are too inconsistent: CV at 0.15 or above, or a score range above 2 points. Those essays go to a human grader before being used.
GPT passes 55.4% of essays straight through without review. Qwen flags 92.2% of essays, meaning a human needs to check almost every Qwen score before it can be used.
The radar chart compares both models across five dimensions where higher is better. No model wins everywhere. Which one to use depends on what your pipeline needs most.