đź“„ Tech Report: https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf | đź“Š Data: https://huggingface.co/datasets/allenai/tutormoments-preview | đź’» Code: https://github.com/allenai/tutormoments
Today we’re introducing a preview of TutorMoments, a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work.
TutorMoments is a replay-based evaluation built off real one-on-one math tutoring sessions. Experienced math teachers go through transcripts collected from a U.S. tutoring program and flag the moments where a tutor had to choose between making a problem easier to get started on and pushing the student to do more of the reasoning themselves. TutorMoments then takes the transcript up to that decision point, hands it to a language model, and has the model take over as the tutor in a simulated session – with the student played by another language model – to see what the LLM tutor does.
Told only to “tutor well,” we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor’s prompt improves performance, but it doesn’t close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call.
As part of our commitment to open research, we’re releasing a dataset of de-identified tutoring transcripts, the code for running our replay pipeline, and the model tutor replays of the key moments we evaluated in those transcripts for reproducibility. We hope TutorMoments gives educators, researchers, and the teams building AI tutors a sharper way to ask how a model handles the pedagogical decisions that matter most—and helps the field build tutors that adapt to each student instead of doing the work for them.
What makes a good tutor?
Ask a good math tutor for help and you’ll likely get a question back like, “What do you know about what the problem is asking?” That isn’t unhelpfulness–part of strong teaching is diagnosing what students do know and providing the right support for them in the moment. Immediately volunteering support would rob a student of the intellectual work that helps them learn. Sometimes support is needed; other times what’s most effective is a push to solidify understanding by explaining a correct answer.
Language models, though, are trained to be helpful, and a helpful assistant tends to do the hard part for you—explaining the concept, laying out the steps, and guiding you to the answer. In a tutoring session, that can cut short the productive struggle—the effortful, sometimes frustrating problem-solving that learning research has long tied to stronger understanding.
Most benchmarks for language models acting as tutors don’t capture this tension. They tend to reward one behavior in particular – never giving away the answer to a problem, say, or always offering a hint – without accounting for whether that was the right move for where the student actually was in their understanding. But good tutoring isn’t a single fixed behavior you can identify across the board. It’s a judgment call: what does this student need, right now, on this problem?
How TutorMoments works
TutorMoments is built on real tutoring data. The dataset we’re releasing, TutorMoments-Preview, is 462 de-identified, text-only transcripts of real one-on-one math tutoring with U.S. students in grades 2-7, with more than 1,500 teacher-annotated key moments and several thousand free-text annotations from 27 U.S.-based teacher annotators. The transcripts come from a high-dosage tutoring program whose students mostly attend Title I schools, shared under a research clause agreed to by parents and guardians; all data was stripped of identifying details, first by the provider and then through an additional math-aware pipeline.
All annotations came from experienced math teachers, whom we asked to read the transcripts and mark key learning moments—noting what was going on, what the tutor did, and how it landed for the student. Each key moment is a decision point where the tutor had to weigh scaffolding (making a problem more accessible) against pushing for rigor (encouraging the student to do harder thinking).
TutorMoments runs by pausing a transcript at one of those key moments and handing the session to a language model, which takes over as the tutor for five turns with a simulated student. We call each of these model-generated continuations a replay. An LLM-based scoring pipeline then rates each replay on three things: whether the model (1) scaffolded when the student needed support, (2) pushed for rigor when the student was ready for more challenge, and (3) avoided over-scaffolding (reducing the challenge more than the moment called for).
The scoring pipeline starts from a teacher-defined ground truth: for each key moment, whether it called for scaffolding or for a push for rigor. Several teachers annotated each moment, and when they disagreed we took the majority label—if three teachers annotated a moment and two called for rigor while one called for scaffolding, the ground truth is rigor. A separate LM classifier validated against teacher annotations then decides whether the tutor’s actual move matches what the moment called for—an “appropriate” turn means the tutor’s classified action (scaffold, push for rigor, or over-scaffold) lines up with what teachers judged the moment to call for.
Preliminary results
We ran seven LLMs through TutorMoments using two prompts: a plain prompt that gives no real guidance – it only tells the model to use what it knows about good tutoring to respond to the student – and an evaluation-aware prompt that spells out the trade-off between scaffolding, over-scaffolding, and pushing for rigor. Each model was scored over key moments drawn from the tutoring transcripts, split evenly between moments where scaffolding was the right approach and moments that called for rigor.
Every number in the table is a rating between 0 and 1 – the share of the relevant moments where the model did the appropriate thing – so a higher score means the model made the right call more often. A 0.50 on appropriate rigor, for instance, means the model pushed for rigor in half of the moments that called for it.
A few things to keep in mind when reading the scores:
Human tutors are a naturalistic reference, not a ceiling. We don’t treat human tutors as a model of ideal practice—even experienced tutors make less-than-optimal choices in the moment. Scored the same way at the same decision points, the human tutors in our transcripts get 0.458 (appropriate scaffolding), 0.182 (appropriate rigor), and 0.496 (avoids over-scaffolding)—all below the models’ evaluation-aware scores and around the range of their plain-prompt scores. But this isn’t a claim that AI tutors outperform human teachers. Annotators specifically looked for moments where tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice.
The scores measure tutor behavior, not learning. Replays use a simulated “oracle” student, so the numbers reflect how a model acts at a decision point—not whether a real student learned.
Rigor is noisier than scaffolding. The scoring pipeline detects rigor pushes less reliably, and there are fewer rigor moments (260) than scaffolding moments (738) in the underlying annotations.
The clearest pattern in the table is how much the prompt matters: every model scores higher under the evaluation-aware prompt than under the plain one. That suggests a model’s default “helpful assistant” behavior isn’t enough on its own to tutor well. But spelling out the trade-off in the prompt only goes so far—while it lifts every score, models still differ widely in how they interpret the enhanced prompt and even the best scorers have plenty of room to improve.
We also break down the moves that tutors made under each scenario. While prompting encourages models to push for rigor, they use fewer strategies than humans do, often relying on asking students to explain their answers. In contrast, human tutors employ more varied strategies and are much more likely to step back and let students work independently.
Limitations and next steps
TutorMoments is still early in its development, and it has several limitations at this stage. The biggest is that automated evaluation gives us signal about how a model behaves at a decision point, but it can’t stand in for studies with real students and real learning outcomes. The dataset is also narrow: U.S.-based, mostly elementary and middle-school math, annotated by a single pool of educators. Our findings may not generalize to other subjects, grade levels, or settings.
We’re sharing this preview to gather feedback as we build toward a larger, multimodal dataset, a stronger scoring pipeline, and deeper analysis.
Acknowledgments
This project has been made possible in part through support from the Gates Foundation and Learning Commons.