TutorMoments tests whether AI tutors know when to hold back
A team behind the TutorMoments project has released a preview of a new evaluation framework built to test one of the hardest calls in teaching: whether a language model acting as a tutor knows when to step in and help a student and when to hold back and let the student do more of the reasoning. The framework is replay based. Experienced math teachers first go through transcripts from real one-on-one tutoring sessions and flag the moments where a human tutor had to choose between making a problem easier and pushing the student to think harder. TutorMoments then hands the transcript, up to that decision point, to a language model, which takes over as the tutor for five turns against a simulated student played by another model.
The underlying dataset, TutorMoments-Preview, contains 462 de-identified, text-only transcripts of real one-on-one math tutoring with U.S. students in grades 2 to 7, drawn from a high-dosage tutoring program whose students mostly attend Title I schools. The data was shared under a research clause agreed to by parents and guardians and stripped of identifying details by the provider and then by an additional math-aware pipeline. Twenty-seven U.S.-based teacher annotators produced more than 1,500 teacher-annotated key moments and several thousand free-text annotations. For each key moment, the ground truth is set by majority vote among the teachers who annotated it: if three teachers judged a moment and two called for pushing rigor while one called for scaffolding, the ground truth for that moment is rigor. A separate LM classifier, validated against the teacher annotations, then judges whether the tutoring model's actual move, classified as scaffold, push for rigor, or over-scaffold, matches what the moment called for.
In the preliminary results, seven LLMs were run through TutorMoments under two prompt conditions: a plain prompt that offers no real guidance beyond telling the model to tutor well, and an evaluation-aware prompt that spells out the trade-off between scaffolding, over-scaffolding, and pushing for rigor. Every model scored higher under the evaluation-aware prompt than under the plain one, showing that a model's default helpful-assistant behavior is not enough on its own to tutor well. But spelling out the trade-off only closes part of the gap: models still differ widely in how they interpret the enhanced prompt, and even the best scorers have room to improve. As a naturalistic reference point, not a ceiling, human tutors in the transcripts were scored the same way and got 0.458 on appropriate scaffolding, 0.182 on appropriate rigor, and 0.496 on avoiding over-scaffolding, all below the models' evaluation-aware scores and roughly in the range of their plain-prompt scores. The authors caution this is not a claim that AI tutors outperform human teachers, since annotators specifically looked for moments where the tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice. Rigor moments proved noisier and rarer than scaffolding moments in the underlying annotations, 260 versus 738, and the scoring pipeline detects rigor pushes less reliably. When models did push for rigor, they relied on fewer distinct strategies than human tutors, most often asking the student to explain their answer, while human tutors used more varied strategies and were far more likely to step back and let the student work independently.
Alongside the preview, the team released the TutorMoments-Preview dataset, the code for running the replay pipeline, and the model tutor replays of the key moments evaluated, for reproducibility. The authors describe TutorMoments as still early: automated evaluation can show how a model behaves at a decision point but cannot substitute for studies with real students and real learning outcomes, and the dataset itself is narrow, U.S.-based, mostly elementary and middle-school math, and annotated by a single pool of educators, so the findings may not generalize to other subjects, grade levels, or settings. The project was made possible in part through support from the Gates Foundation and Learning Commons, and the team says it is sharing the preview to gather feedback while it builds toward a larger, multimodal dataset, a stronger scoring pipeline, and deeper analysis.
Key facts
- TutorMoments-Preview contains 462 de-identified transcripts of real one-on-one math tutoring with U.S. students in grades 2 to 7, with more than 1,500 teacher-annotated key moments from 27 U.S.-based teacher annotators.
- Seven LLMs were tested under two prompts; every model scored higher on the evaluation-aware prompt, which spells out the scaffolding-versus-rigor trade-off, than on the plain prompt.
- Scored the same way as the models, human tutors got 0.458 on appropriate scaffolding, 0.182 on appropriate rigor, and 0.496 on avoiding over-scaffolding, below the models' evaluation-aware scores and close to their plain-prompt scores.
- Rigor decision points are rarer and noisier in the data than scaffolding ones, 260 versus 738 annotated moments, and models leaned on fewer distinct strategies to push rigor than human tutors did.
- The dataset, the replay pipeline code, and the model tutor replays were released for reproducibility; the work was supported in part by the Gates Foundation and Learning Commons.
Why it matters
Most benchmarks that score language models as tutors reward one fixed behavior across the board, such as never giving away the answer or always offering a hint, without asking whether that move fit the student's actual moment of understanding. TutorMoments targets that gap directly: good tutoring is a judgment call between scaffolding a problem down and pushing a student toward harder, independent reasoning, and a model trained to be maximally helpful tends to short-circuit that judgment by doing the hard part for the student. The preliminary results back that concern up: told only to tutor well, models over-help and rarely push for deeper thinking, and even a prompt that spells out the trade-off only partly closes the gap to how the moment actually calls to be handled.
Who it affects
The release targets educators evaluating AI tutoring tools, researchers studying how language models handle pedagogical trade-offs, and teams building AI tutors who need a way to check whether a model adapts to a student rather than just answering for them. The tutoring transcripts themselves come from students in grades 2 to 7 in a U.S. high-dosage program serving mostly Title I schools, though TutorMoments evaluates model behavior on replays rather than measuring outcomes for those students directly.
How to use it
The team released the TutorMoments-Preview dataset of 462 de-identified transcripts, the code for running the replay pipeline, and the model tutor replays of the key moments they evaluated, so others can reproduce the results or run their own models through the same evaluation. A model is tested by taking over a transcript at a flagged decision point and playing tutor for five turns against a simulated student, then scoring whether it scaffolded when needed, pushed for rigor when the student was ready, and avoided over-scaffolding.
How solid is it
The ground truth for each key moment comes from majority agreement among the experienced math teachers who annotated it, and a separate LM classifier that scores the tutor's actual move was itself validated against those teacher annotations. Human tutors were run through the same scoring as a naturalistic reference point rather than a ceiling, since even experienced tutors make less-than-optimal choices and the annotators specifically hunted for moments where the tutoring could have gone better. The authors are explicit that the scores measure a model's behavior at a decision point using a simulated oracle student, not whether a real student actually learned.
Risks and caveats
The authors flag the evaluation as early and the dataset as narrow: it is U.S.-based, mostly elementary and middle-school math, and annotated by a single pool of educators, so the findings may not generalize to other subjects, grade levels, or settings. Rigor is measured less reliably than scaffolding because rigor moments are rarer in the annotations (260 versus 738) and the scoring pipeline detects rigor pushes less consistently. Automated replay evaluation can show how a model acts at a decision point but cannot substitute for studies that track real students and real learning outcomes.
“Told only to "tutor well," we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking.”
— the authors