Finishing the task and learning the skill are different outcomes

Microsoft’s October 26 introduction to Learning Curves, a collection of essays from Microsoft AI Economy Institute fellows, argues for context-sensitive, human-centered education. It is a timely contribution from a company that sells AI tools, not a new controlled test of learning.

The underlying question is practical: did a student produce a better answer with assistance, or become better able to solve the problem independently? Those are different achievements.

Two studies published in 2025 make the distinction concrete. They test particular teaching arrangements, not whether AI is universally good or bad for education.

What happened when the chatbot was removed

A PNAS study by Hamsa Bastani and colleagues examined nearly 1,000 mathematics students at one Turkish high school. During four sessions, classes used either no AI, a GPT-4 chat interface called GPT Base, or GPT Tutor, which incorporated teacher-designed solutions and hints.

Both AI groups did better during assisted practice. But on the subsequent closed-book, closed-laptop exam, the GPT Base group scored about 17% lower than the comparison group. That is a relative difference in scores, not a fall of 17 percentage points.

GPT Tutor’s unassisted exam performance was not statistically distinguishable from the comparison group. Its safeguards largely avoided the measured harm; the result did not establish an independent-learning gain.

The experiment measured short-term performance within those sessions. One school, one subject and a brief intervention do not establish a universal or long-term effect.

A structured tutor is a different intervention

A Scientific Reports study by Greg Kestin and colleagues included 194 eligible students in an introductory Harvard physics course. In a two-lesson crossover design, students experienced both a custom AI tutor and in-class active learning, in different weeks.

The AI-tutored condition produced better immediate post-test results. But the intervention was more than an unrestricted chat window: it used instructor-written material, worked solutions, videos and a platform that guided students through problem parts.

The authors do not claim their approach will always outperform classroom learning, especially for more complex synthesis or critical thinking. Two lessons also cannot settle whether students retain the gains across a whole course.

Read the assessment before comparing the claims

These trials have different students, subjects, tools and comparison groups. One compares chatbot-supported practice with no-AI practice; the other compares a designed tutoring lesson with an active classroom. Their results cannot be turned into a single league table of AI tutors.

The useful reading questions are straightforward. Was assistance available during the test? Was the assessment immediate or delayed? Did students face a new problem rather than repeat a supplied answer? What did the teacher and software contribute besides the model?

That is a way to interpret evidence, not a promise that one prompt or product will produce a particular result. A tool’s popularity, a completed worksheet and a student’s feeling of confidence are not substitutes for measuring what the student can do afterwards.

Sources

  1. Microsoft: Learning CurvesOctober 26 Microsoft essay introduces Learning Curves and identifies Microsoft AI Economy Institute fellows. Current-event peg and sponsor perspective only; book not purchased or represented as a new classroom trial.
  2. Bastani et al.: high-school mathematics trialBastani et al., PNAS, published June 25, 2025. Full original article read through Europe PMC XML because publisher/PMC web paths showed challenges; DOI identity checked. Nearly 1,000 students at one Turkish high school; assisted practice versus same-session unassisted exam, relative 17% decline for GPT Base, GPT Tutor not significantly different from control. No long-term causal estimate or reanalysis claimed.
  3. Kestin et al.: structured AI tutoring trialKestin et al., Scientific Reports, published June 3, 2025. Full original article read through Europe PMC XML, DOI matched. 194 eligible Harvard physics students, two-lesson crossover, structured tutor versus active learning, immediate post-tests and context limits. No general replacement-of-teachers or long-term retention claim.