Norbert Noster, Sebastian Gerber, Hans-Stefan Siller
9 min
This study investigates how pre-service mathematics teachers interact with ChatGPT when solving mathematical tasks. Using the KTMT (Knowledge for Teaching Mathematics with Technology) model, the researchers examined participants' ability to handle the model's 'black box' nature—specifically its tendency to produce plausible but incorrect mathematical information. The authors analyze how these future teachers use prompts, evaluate the model's accuracy, and perceive their own technological and mathematical competence.
Eleven pre-service teachers were tasked with solving four distinct mathematical problems using ChatGPT-3.5: determining a greatest common divisor, identifying the count of 'zebra numbers' (a defined combinatorial task), and two probability problems involving coin tosses and urns. Researchers utilized qualitative content analysis on the resulting chatlogs and follow-up interviews to code for prompting strategies, error detection behavior, and self-evaluation of mathematical/technological skills.
The findings highlight a concerning gap in teacher competence. While participants used various prompting strategies, such as zero-shot (direct task submission) and few-shot (providing definitions/examples) techniques, these were largely spontaneous rather than systematic. Critically, participants frequently failed to identify incorrect answers generated by the chatbot. When the model provided erroneous information, a significant number of pre-service teachers adopted these incorrect answers, suggesting that their mathematical judgment was overwhelmed by the technology's authoritative output. Furthermore, participant self-evaluations did not align well with their actual performance, as many felt confident but still failed to reach the correct solutions.
As large language models become standard tools, mathematics educators must move beyond viewing them as simple search engines. This research demonstrates that without explicit training in prompt engineering and, more importantly, in verifying AI-generated mathematical content, teachers risk propagating inaccuracies. The study underscores the urgent need to integrate critical AI literacy into teacher education programs, moving toward a deeper understanding of 'mathematical fidelity'—the ability to judge when and how a digital tool can be trusted.
Abstract The use of large language models like ChatGPT is widely discussed for educational purposes. Using this technology requires teachers to have appropriate competences that incorporate knowledge of how to make use of this technology. In this study, we investigate pre-service teachers’ knowledge through the lens of the KTMT model (“Knowledge for Teaching Mathematics with Technology” model), a domain-specific variant of the TPACK-model. One component is represented in mathematical fidelity as knowledge of the mathematical accuracy of the technology, which in case of large language models is of special interest, as it may produce erroneous but plausible-sounding information. Furthermore, prompting techniques are of interest as technological knowledge, which influence mathematical fidelity. For this study, eleven pre-service teachers were asked to solve four different mathematical tasks with the help of ChatGPT. The chatlogs and information provided in an interview after working on the tasks are analyzed using qualitative content analysis. Results show that both correct and incorrect answers were produced for all tasks. The rate of pre-service teachers providing an incorrect answer is high when having been presented with an incorrect answer generated by the large language model. Despite having access to ChatGPT as a tool, many of the participants were not able to provide correct answers to all tasks. Furthermore, the mathematical fidelity was often over- and, in some cases, underrated. The mathematical knowledge seems to have changed while working with ChatGPT. Based on the applied prompting techniques, the pre-service teachers showed a deficiency in technological knowledge.
Sam: The authors chose tasks requiring basic mathematics relevant to lower secondary teaching, but with different expected difficulties for the chatbot. They included a greatest-common-divisor exercise, a counting problem, and probability problems involving coins and an urn. The counting problem was the distinctive one.
Alex: Walk us through that example. What did participants have to count?
Sam: They were given a definition of a zebra number. It uses only two different digits, with identical digits never directly adjacent. For a three-digit number, that means the first and last digits match, while the middle digit differs.
Alex: And a leading zero would mean it wasn’t a three-digit number. But zero could still appear in the middle.
Sam: That distinction is central. The correct count is eighty-one, allowing zero in the middle but excluding the repeated outer digit. Because “zebra number” is unfamiliar, solving the task requires applying the supplied definition rather than relying on a familiar term.
Alex: That also sounds like a useful test of prompting. Did participants just paste in the task, or try to establish what the chatbot understood?
Sam: Both approaches appeared. The authors call a direct request without related preparation zero-shot prompting. They also identified few-shot prompting, where participants supplied related definitions or examples before asking for the solution. That preparatory approach was concentrated in the zebra task.
Alex: So the task seemed to change the interaction. Participants were more likely to check an unfamiliar definition than a familiar calculation.
Sam: That is the authors’ interpretation, but they treat the dependency as a hypothesis. Some participants also requested feedback on their own reasoning. Others repeated questions, asked for elaboration, changed languages, or used the chatbot as a calculator.
Alex: Asking for feedback sounds sensible, but it also hands the model authority over your answer. Did the conversations show how that could become difficult?
Sam: One participant described the alternating-digit structure correctly, but allowed any digit in the middle. Their reasoning overlooked that the middle digit must differ from the outer digit. They asked the chatbot to correct or help with that reasoning.
Alex: Then the useful outcome depends on whether the feedback actually identifies the mistake. A detailed prompt alone doesn’t establish that the response will be dependable.
Sam: And the study does not show that the preparatory prompts reliably fixed the problem. The authors say these often tested terms through examples rather than providing systematic preparation. More elaborate prompting techniques were not evident, while repeated questions were common enough to concern them.
Alex: Let’s separate the model’s performance from the participants’ performance. What happened on that zebra task?
Sam: Every participant encountered an incorrect chatbot answer on it. Some also encountered correct answers. Five participants ultimately gave an incorrect solution themselves, making this the task with the most wrong final answers.
Alex: Seeing both correct and incorrect responses creates a selection problem. The user still has to decide which explanation deserves acceptance.
Sam: That’s the central difficulty. Across all tasks, the chatbot produced both correct and incorrect responses. For coding purposes, an answer counted as incorrect if its result or reasoning was wrong, or if it contained contradictions. A correct final number alone was not sufficient.
Alex: Did the future teachers notice errors, even when their final answers were wrong?
Sam: Several did, especially on the zebra task. They challenged examples with leading zeros, missing middle zeros, or mismatched outer digits. But asking again or requesting more detail was less decisive evidence that someone had identified a mathematical error.
Alex: Was exposure to wrong chatbot answers consistently associated with wrong participant answers, or did that vary across tasks?
Sam: It varied. On the urn problem, almost everyone answered correctly despite many encountering erroneous responses. The only statistically significant association appeared on the coin-toss task: more incorrect chatbot messages were associated with a greater likelihood of a wrong participant answer.
Alex: That isn’t enough to say the chatbot made them worse. Did the researchers measure how these same participants performed without it?
Sam: No. They asked participants to assess their unaided ability, but did not observe an unaided comparison. The authors leave open whether wrong answers reflected insufficient mathematical knowledge, difficulty using that knowledge during the interaction, or influence from the chatbot.
Alex: What about trust? A brief encounter with unreliable answers might change expectations, even if it doesn’t improve mathematical performance.
Sam: The interviews suggest that happened for some participants. Some initially high expectations fell, while some moderate expectations rose. The authors interpret this as movement towards a more differentiated view of the model’s mathematical capabilities, not simply greater distrust.
Alex: Before translating that into teacher education, what is the biggest constraint on the evidence?
Sam: The sample was small and self-selected from a seminar about chatbots in mathematics instruction. The authors also could not determine whether task order influenced behaviour. I’d treat this as evidence for questions worth investigating, not a population estimate or a demonstrated training effect.
Alex: Given those limits, who should read the full document, and where should they start?
Sam: Teacher educators and researchers studying human–AI mathematical reasoning should read it. Start with the task descriptions, then the findings on correctness of chatbot and participant answers. That comparison shows why evaluating the model alone misses the user’s challenge of choosing between competing responses.
Alex: And what should everyone else retain, without turning this small study into a blanket verdict against chatbots?
Sam: The authors propose varied hands-on tasks and reflection as a route towards better-informed teacher preparation. They have not yet established the effectiveness of that route. The line to remember is: a chatbot can offer an answer, but the user still has to judge the mathematics.
Alex: Keep that distinction in mind.