Two AI Tutor Studies, Opposite Results. The Tool Wasn’t the Point.

Every few weeks a new headline tells us whether AI tutors in schools are the answer or the problem. This week Jill Barshay at The Hechinger Report gave us something more useful: two studies from the same researcher with opposite results. One AI tutor was linked to higher math scores for middle schoolers. The other, in college courses, was linked to lower grades and less engagement.

Same researcher. Same general idea. Opposite outcomes.

Here’s the thing… if you only read one of those studies, you could write a very confident headline either way. Read them together and the story isn’t about the tool at all. It’s about design.

What the two AI tutor studies found

Jing Liu directs the Center for Educational Data Science and Innovation at the University of Maryland. According to Barshay’s Proof Points column, Liu wanted the two studies released together precisely to show that AI in education is neither all good nor all bad. I appreciate that. It’s rare to see a researcher put a positive and a negative result side by side on purpose.

Study one: a middle school math tutor. About 6,000 middle schoolers in Deer Valley, Arizona, used an AI math tutor called Goblins during the 2025-26 school year. It offered step-by-step help through text chat, a virtual whiteboard and spoken conversation, and could work in Spanish and six other languages. Students who solved at least two problems a week scored significantly higher on a year-end math test than similar students who used it less or not at all. Accelerate, the nonprofit that commissioned the study, estimated the 0.11 standard deviation difference at roughly three extra months of learning.

Important caveats, and the researchers say them out loud: students weren’t randomly assigned, so more motivated students may have both practiced more and scored higher. Accelerate is planning a randomized controlled trial. And only 6% of students used the tool for at least half an hour a week. Most of the use came through work teachers assigned.

Study two: a college course assistant. Before rolling out an AI course assistant to about 40,000 students, University of Maryland Provost Jennifer King Rice asked Liu to test it. In fall 2025, 15 of 30 professors were randomly assigned to add the assistant to their courses. It was available 24/7 inside Canvas and trained on the syllabus, readings and assignments. The result, written up in a randomized trial posted to EdWorkingPapers: about 1,100 students with the assistant earned lower grades than about 1,200 students without it, by an average of 0.11 points on a four-point scale. They also engaged less with course materials and were less likely to go to office hours or meet with instructors and peers. The negative effects were larger for first-generation students.

Only 15% of students in those classes used the assistant, and most asked it for answers rather than tutoring. Liu and Rice suspect the tool’s constant presence signaled that AI use was encouraged, which may have pushed students toward outside chatbots too. They are clear that they can’t confirm that.

AI tools “could exacerbate inequities in learning outcomes.”

Jennifer King Rice, University of Maryland provost, quoted in The Hechinger Report

Let’s give credit where it’s due. Maryland tested the tool before rolling it out to everyone, and then made the bad news public. That’s exactly what we should want from every school and university buying AI right now.

Here’s what the headline leaves out

If you’re someone who believes AI is making students lazy, the college study looks like proof. And I want to be fair here: it’s a randomized trial, it’s a real negative result, and the finding that first-generation students were hurt most should worry all of us. I’m not going to wave that away.

If you’re someone who believes AI tutors will finally personalize learning for every kid, the middle school study looks like proof. But it’s correlational, and most students barely used the tool.

Neither camp gets to declare victory. Also worth saying: these are different students in different settings, middle school math and college courses, so this isn’t a clean head-to-head comparison. What we can do is look at what was different about how each tool showed up in students’ lives.

In one, teachers decided when and how to use it, assigned the problems, and the district recommended it as a supplement, not the core. In the other, the tool was simply there, all day and all night, next to every assignment, and students were mostly left to figure out what it was for. So they used it for what most of us would use it for when no one has taught us otherwise… getting the answer.

That’s not a technology finding. That’s a teaching and design finding. I wrote about this back in 2009 in What’s the purpose of going 1:1, when schools were handing out laptops and hoping learning would change on its own. It didn’t then. Handing out an AI tutor and hoping won’t work now either.

This is why I keep saying, and why I wrote Human Still Required for school leaders: AI didn’t break school — it exposed it. If our courses already let students get by on answers, an always-on answer machine will expose that fast. The tool can explain, summarize and give feedback. Deciding what the learning is for, and designing the task so students actually have to think, is still human work.

The question is the skill

A second piece in The Hechinger Report this week connects right to this. Hiranya K. Nath, an economics professor at Sam Houston State University, wrote a Teacher Voice essay on teaching students to ask better questions. He asked his students to act as financial journalists and write three questions for a senior Federal Reserve official, and told them he would grade the questions, not the answers. Students who were comfortable explaining concepts hesitated. Others thought harder than they had all semester.

His point is that AI’s output depends on the quality of the questions we bring to it, so asking good questions becomes more valuable, not less. Two lines from his essay stuck with me:

“The tool does not define the problem. We do.” … “We are preparing students for a world of answers while they are entering a world of questions.”

Hiranya K. Nath, The Hechinger Report

Put that next to the Maryland result. Students who asked the assistant for answers got answers, and grades went down. That’s not a surprise. It’s what happens when we never teach the difference between asking for an answer and asking for help understanding.

Preparing students for their future, not our past, means teaching that difference on purpose. Their future includes AI tutors, AI assistants and AI everything. Banning the tools doesn’t teach that skill. Leaving them lying around doesn’t either. Teaching does.

AI tutors in schools: what to do this week

• Ask who decides when the tool is used. In the study with positive results, teachers assigned the work. If your AI tutor is just “available,” ask what students are actually using it for.
• Pilot before you scale, and measure it. Maryland tested before a 40,000-student rollout. Pick a few classes, compare, and be willing to share what you find, good or bad.
• Look at who is helped and who is hurt. The college study found bigger negative effects for first-generation students. Break your own data down by student group before you call anything a success.
• Teach the “help, not answers” move explicitly. Model prompts like “Don’t give me the answer, ask me a question that helps me find my mistake.” Have students reflect on how they used the tool.
• Grade a question. Borrow Nath’s idea. Once this week, assess the questions students write instead of the answers they give.

Frequently asked questions

Do AI tutors improve student learning?

The evidence is mixed. A study of about 6,000 Arizona middle schoolers linked regular use of an AI math tutor to higher test scores, but it can’t prove the tool caused the gains. A randomized trial at the University of Maryland found an always-available AI course assistant was linked to lower grades and less engagement.

Why would an AI tutor make grades go down?

In the Maryland study, most students who used the assistant asked it for answers rather than tutoring, and students engaged less with course materials and office hours. The researchers suspect the tool’s constant presence signaled that AI use was encouraged, but say they can’t confirm that.

Should my school stop using AI tutors?

Not necessarily. These studies suggest the design matters more than the tool: who assigns the work, what the tool is for, and whether students are taught to use it for understanding. Pilot carefully, measure results by student group, and adjust.

The tool was never the point

We’ve had this conversation with every new technology I’ve seen come into schools in my 27 years in education. Interactive whiteboards, laptops, tablets, now AI. The tool gets the headline. The design decides the result.

I’d love to see more schools do what Maryland did: test, measure, and share the results even when they’re uncomfortable. If your school or district is trying to figure out where AI fits, this is the work I do with leaders every week. You can find me at JeffUtecht.com.

So here’s my question for you: if you looked at how AI tools are showing up in your classrooms right now, would you say they’re designed in, or just left lying around?

Sources

• Same researcher, opposite results — Jill Barshay, The Hechinger Report (Proof Points), November 2, 2026
• The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial — Jing Liu, Jennifer King Rice et al., EdWorkingPapers, October 2026
• TEACHER VOICE: In the age of AI, we are teaching the wrong skills. Let’s shift to teaching them more about how to think — Hiranya K. Nath, The Hechinger Report, November 2, 2026