A new study of more than 6,000 middle school students in Tennessee suggests that AI tutors can improve learning - but only when the software forces students to slow down and prove they've mastered a skill before moving on.
Students who used an AI tutor combined with a "mastery learning" requirement scored about 3 percentage points higher on a retention test than students who received conventional computerized instruction. The study tested four approaches to practicing fractions: with and without AI tutoring, and with and without a requirement to answer three consecutive practice questions correctly after making a mistake.
The winning combination was AI tutoring plus repetition. Students in that group spent more time per question than students in any other group, and they were more likely to get the next question right after making an error.
Why the AI tutor made a difference
The researchers believe the AI helped because it walked students through their specific mistakes instead of simply showing them a worked example. Without AI, students could view a step-by-step solution after getting a problem wrong - but they could skim past it without understanding what went wrong.
The AI tutor responded directly to each student's work and guided them through the error. That interaction appeared to increase engagement, though the advantage showed up mainly on the easiest fraction questions - the ones most similar to what students had practiced. The benefit did not extend to more challenging problems.
The study, "Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment," was conducted by researchers from the University of Toronto and the University of Pennsylvania's Wharton School. A working paper is scheduled to be circulated by the National Bureau of Economic Research on Aug. 17 and has not yet been peer-reviewed.
"I don't want to jump out and say we've demonstrated that AI is going to be the game changer that we hope it is," said Philip Oreopoulos, lead author and an economist at the University of Toronto. "But it might be the first kind of evidence that shows there's at least some hints that it has some positive value against no AI at all."
A small effect with a clear direction
The 3-point advantage is modest, and Oreopoulos cautioned against concluding that mastery learning is the best way to use AI in math education. The researchers tested only four combinations in a 50-minute intervention; they didn't measure how the approach might work over months of daily use.
Still, the findings matter because much of the existing evidence on AI in education points the other way. There's mounting research showing AI tools that hand students answers can short-circuit learning. This study offers a counterexample: AI that makes students work through their mistakes can add value.
Requiring three correct answers in a row is common in educational software, but it's not a guarantee of deep understanding. Students can guess their way to mastery or succeed through repetition without truly learning the skill. The study's results suggest the combination of AI guidance and repetition is what helped - not either feature alone.
Why this matters for educators
For teachers weighing whether to use AI tutoring tools, the takeaway is about design: the AI itself isn't the benefit. The benefit comes from how the software handles mistakes and forces follow-through. An AI tutor that simply provides answers is likely to hurt learning. One that diagnoses errors and requires demonstrated mastery before advancing may help - at least for foundational skills like fractions.
Oreopoulos's larger ambition is to keep testing different features against one another, continually improving the software. For now, he said, the goal was more modest: to show that AI "has a little bit of benefit, and talk about its potential." That potential may lie not in helping students learn faster, but in helping them slow down.
For educators exploring how to integrate AI into their classrooms, the study points to practical questions: Does the tool explain mistakes or just show answers? Does it require students to demonstrate the skill again after an error? Those design choices may matter more than whether AI is involved at all. AI for Teachers Courses can help educators evaluate these tools critically, while broader AI for Education resources cover how to assess learning software before bringing it into the classroom.
Your membership also unlocks:
