ResearchPod Summary
This study investigates how pre-service mathematics teachers interact with ChatGPT when solving mathematical tasks. Using the KTMT (Knowledge for Teaching Mathematics with Technology) model, the researchers examined participants' ability to handle the model's 'black box' nature—specifically its tendency to produce plausible but incorrect mathematical information. The authors analyze how these future teachers use prompts, evaluate the model's accuracy, and perceive their own technological and mathematical competence.
Eleven pre-service teachers were tasked with solving four distinct mathematical problems using ChatGPT-3.5: determining a greatest common divisor, identifying the count of 'zebra numbers' (a defined combinatorial task), and two probability problems involving coin tosses and urns. Researchers utilized qualitative content analysis on the resulting chatlogs and follow-up interviews to code for prompting strategies, error detection behavior, and self-evaluation of mathematical/technological skills.
The findings highlight a concerning gap in teacher competence. While participants used various prompting strategies, such as zero-shot (direct task submission) and few-shot (providing definitions/examples) techniques, these were largely spontaneous rather than systematic. Critically, participants frequently failed to identify incorrect answers generated by the chatbot. When the model provided erroneous information, a significant number of pre-service teachers adopted these incorrect answers, suggesting that their mathematical judgment was overwhelmed by the technology's authoritative output. Furthermore, participant self-evaluations did not align well with their actual performance, as many felt confident but still failed to reach the correct solutions.
As large language models become standard tools, mathematics educators must move beyond viewing them as simple search engines. This research demonstrates that without explicit training in prompt engineering and, more importantly, in verifying AI-generated mathematical content, teachers risk propagating inaccuracies. The study underscores the urgent need to integrate critical AI literacy into teacher education programs, moving toward a deeper understanding of 'mathematical fidelity'—the ability to judge when and how a digital tool can be trusted.
Sam: A chatbot’s wrong answer can become a future mathematics teacher’s wrong answer. In a small study, some teachers in training missed ChatGPT’s mistakes, and access to the tool did not ensure correct solutions.
Alex: That matters if they will eventually guide pupils using these tools. Was this a test of the chatbot, or of the people working with it?
Sam: Both, but the focus was future teachers’ knowledge. It’s an exploratory study by Norbert Noster, Sebastian Gerber, and Hans-Stefan Siller, published in Digital Experiences in Mathematics Education. They followed eleven mathematics teachers in training at the University of Würzburg.
Alex: So the practical question is whether knowing mathematics also means knowing when to trust a chatbot.
Sam: That’s the concern for teacher educators. We’ll look at how the authors observed those judgments, why one unusual task exposed difficulties, and what this small study can actually establish.
Alex: Chatbot errors were already a known problem. What does this study add beyond another demonstration that a model gets mathematics wrong?
Sam: It connects those errors to the knowledge teachers need when using technology. The authors use the Knowledge for Teaching Mathematics with Technology framework. That connects mathematical knowledge, knowledge of teaching, and knowledge of the technology.
Alex: So operating the software is only one part. A teacher also needs to understand how it changes the mathematics learners encounter.
Sam: Yes, and a central concept is mathematical fidelity: the mathematical accuracy of a technology. Here, that includes knowing that plausible chatbot explanations can be wrong. The framework also asks what challenges the technology creates for learners.
Alex: How did they make that knowledge visible? Asking people whether they trust artificial intelligence would give a rather incomplete picture.
Sam: They observed participants solving four mathematics tasks with ChatGPT. Pencil and paper were allowed. Immediately afterwards, they interviewed each participant about their answers, previous experience, expectations, and perceived ability to solve the tasks unaided.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: Which model were they using? That matters for how far we can carry these findings forward.
Sam: They used ChatGPT version three point five. The study concerns that model in that setting, not every chatbot available today. The researchers analysed both chatlogs and interview transcripts, categorising prompts, errors, and participants’ judgments.
Alex: What made the tasks informative? If everything were either effortless or impossible, there would be little opportunity to observe useful checking.
Sam: The authors chose tasks requiring basic mathematics relevant to lower secondary teaching, but with different expected difficulties for the chatbot. They included a greatest-common-divisor exercise, a counting problem, and probability problems involving coins and an urn. The counting problem was the distinctive one.
Alex: Walk us through that example. What did participants have to count?
Sam: They were given a definition of a zebra number. It uses only two different digits, with identical digits never directly adjacent. For a three-digit number, that means the first and last digits match, while the middle digit differs.
Alex: And a leading zero would mean it wasn’t a three-digit number. But zero could still appear in the middle.
Sam: That distinction is central. The correct count is eighty-one, allowing zero in the middle but excluding the repeated outer digit. Because “zebra number” is unfamiliar, solving the task requires applying the supplied definition rather than relying on a familiar term.
Alex: That also sounds like a useful test of prompting. Did participants just paste in the task, or try to establish what the chatbot understood?
Sam: Both approaches appeared. The authors call a direct request without related preparation zero-shot prompting. They also identified few-shot prompting, where participants supplied related definitions or examples before asking for the solution. That preparatory approach was concentrated in the zebra task.
Alex: So the task seemed to change the interaction. Participants were more likely to check an unfamiliar definition than a familiar calculation.
Sam: That is the authors’ interpretation, but they treat the dependency as a hypothesis. Some participants also requested feedback on their own reasoning. Others repeated questions, asked for elaboration, changed languages, or used the chatbot as a calculator.
Alex: Asking for feedback sounds sensible, but it also hands the model authority over your answer. Did the conversations show how that could become difficult?
Sam: One participant described the alternating-digit structure correctly, but allowed any digit in the middle. Their reasoning overlooked that the middle digit must differ from the outer digit. They asked the chatbot to correct or help with that reasoning.
Alex: Then the useful outcome depends on whether the feedback actually identifies the mistake. A detailed prompt alone doesn’t establish that the response will be dependable.
Sam: And the study does not show that the preparatory prompts reliably fixed the problem. The authors say these often tested terms through examples rather than providing systematic preparation. More elaborate prompting techniques were not evident, while repeated questions were common enough to concern them.
Alex: Let’s separate the model’s performance from the participants’ performance. What happened on that zebra task?
Sam: Every participant encountered an incorrect chatbot answer on it. Some also encountered correct answers. Five participants ultimately gave an incorrect solution themselves, making this the task with the most wrong final answers.
Alex: Seeing both correct and incorrect responses creates a selection problem. The user still has to decide which explanation deserves acceptance.
Sam: That’s the central difficulty. Across all tasks, the chatbot produced both correct and incorrect responses. For coding purposes, an answer counted as incorrect if its result or reasoning was wrong, or if it contained contradictions. A correct final number alone was not sufficient.
Alex: Did the future teachers notice errors, even when their final answers were wrong?
Sam: Several did, especially on the zebra task. They challenged examples with leading zeros, missing middle zeros, or mismatched outer digits. But asking again or requesting more detail was less decisive evidence that someone had identified a mathematical error.
Alex: Was exposure to wrong chatbot answers consistently associated with wrong participant answers, or did that vary across tasks?
Sam: It varied. On the urn problem, almost everyone answered correctly despite many encountering erroneous responses. The only statistically significant association appeared on the coin-toss task: more incorrect chatbot messages were associated with a greater likelihood of a wrong participant answer.
Alex: That isn’t enough to say the chatbot made them worse. Did the researchers measure how these same participants performed without it?
Sam: No. They asked participants to assess their unaided ability, but did not observe an unaided comparison. The authors leave open whether wrong answers reflected insufficient mathematical knowledge, difficulty using that knowledge during the interaction, or influence from the chatbot.
Alex: What about trust? A brief encounter with unreliable answers might change expectations, even if it doesn’t improve mathematical performance.
Sam: The interviews suggest that happened for some participants. Some initially high expectations fell, while some moderate expectations rose. The authors interpret this as movement towards a more differentiated view of the model’s mathematical capabilities, not simply greater distrust.
Alex: Before translating that into teacher education, what is the biggest constraint on the evidence?
Sam: The sample was small and self-selected from a seminar about chatbots in mathematics instruction. The authors also could not determine whether task order influenced behaviour. I’d treat this as evidence for questions worth investigating, not a population estimate or a demonstrated training effect.
Alex: Given those limits, who should read the full document, and where should they start?
Sam: Teacher educators and researchers studying human–AI mathematical reasoning should read it. Start with the task descriptions, then the findings on correctness of chatbot and participant answers. That comparison shows why evaluating the model alone misses the user’s challenge of choosing between competing responses.
Alex: And what should everyone else retain, without turning this small study into a blanket verdict against chatbots?
Sam: The authors propose varied hands-on tasks and reflection as a route towards better-informed teacher preparation. They have not yet established the effectiveness of that route. The line to remember is: a chatbot can offer an answer, but the user still has to judge the mathematics.
Alex: Keep that distinction in mind.