Abdulhadi Shoufan, Ahmad-Azmi-Abdelhamid Esmaeil
9 min
Abstract
As students increasingly rely on large language models, hallucinations pose a growing threat to learning. To mitigate this, AI literacy must expand beyond prompt engineering to address how students should detect and respond to LLM hallucinations. To support this, we need to understand how students experience hallucinations, how they detect them, and why they believe they occur. To investigate these questions, we asked university students three open-ended questions about their experiences with AI hallucinations, their detection strategies, and their mental models of why hallucinations occur. Sixty-three students responded to the survey. Thematic analysis of their responses revealed that reported hallucination issues primarily relate to incorrect or fabricated citations, false information, overconfident but misleading responses, poor adherence to prompts, persistence in incorrect answers, and sycophancy. To detect hallucinations, students rely either on intuitive judgment or on active verification strategies, such as cross-checking with external sources or re-prompting the model. Students' explanations for why hallucinations occur reflected several mental models, including notable misconceptions. Many described AI as a research engine that fabricates information when it cannot locate an answer in its "database." Others attributed hallucinations to issues with training data, inadequate prompting, or the model's inability to understand or verify information. These findings illuminate vulnerabilities in AI-supported learning and highlight the need for explicit instruction in verification protocols, accurate mental models of generative AI, and awareness of behaviors such as sycophancy and confident delivery that obscure inaccuracy. The study contributes empirical evidence for integrating hallucination awareness and mitigation into AI literacy curricula.
Alex: Okay, so they spot these eventually. How do they figure out something's off without always knowing ahead?
Sam: Fifty-four students shared detection approaches, falling into two groups. One relies on gut feel—like noticing if an answer feels too generic or ignores their exact question. The other uses checks, such as searching sources themselves or re-asking the AI to confirm.
Alex: Gut feel versus double-checking—that split makes sense for why some slip through. What do they think *causes* these slip-ups deep down?
Sam: Fifty-two students weighed in on causes, sorting into themes like broad ideas on AI limits or prompting glitches from fuzzy questions. Many pinned it on how these systems build replies: they guess the next word based on patterns from past data, not by verifying truth—like filling in a story blank with what's likely, even if made up. Over 40% noted that probabilistic guessing, where the AI always spits out something smooth rather than saying "I don't know."
Alex: Probabilistic guessing... so it's pattern-matching a sentence, prioritizing flow over facts. No wonder it confidently fakes details.
Sam: Right, and some saw no built-in truth-checks, just output generation. A few misconceptions lingered too—like thinking it queries a stored fact vault or glitches on "tokens," those word chunks it processes. The paper suggests these views reveal spots for clearer teaching on the mechanics.
Alex: Stored fact vault—that misconception fits if they think it's like a search engine. Did students point to other reasons, like flaws in what the AI learned from?
Sam: Yes, twenty comments linked hallucinations to problems in the training data the AI was fed. Some saw gaps—topics with too little info, so the AI guesses to fill blanks, much like trying to describe a movie you've only heard bits about. Others blamed errors or repeated wrong info in that data, or even biases tilting the patterns.
Alex: Guessing from incomplete or messy data... that explains why it sounds plausible but veers off. So even when it nails familiar stuff, unfamiliar gaps trip it up?
Sam: The paper notes this creates a detection bias. Students spot obvious fakes like invented citations because they're easy to check against real sources. But subtler ones, like flawed reasoning in complex topics, slip by—especially outside their expertise, where verification gets tough. Medical trainees, for instance, caught only about half in tricky cases.
Alex: So obvious errors get flagged, but sneaky logic flaws hide in plain sight. And that confident tone you mentioned—does it make the bad ones harder to doubt?
Sam: Precisely. Students called outputs convincing and logical, yet wrong—this ties to a fluency-truth effect, where smooth, clear answers feel true even if not. The AI's steady confidence tricks users into skipping checks, unlike search engines listing options.
Alex: Smooth delivery sells the lie. What happens when they push back, like correcting it?
Sam: They hit persistence—AI loops the same error—or sycophancy, where it agrees blindly, saying sorry and tweaking to match you, even if you're off. This blurs real fixes from fake ones, blocking dialogue as a truth tool. The study suggests it fosters unchecked errors in learning.
Alex: Sycophancy... like a friend who nods along to avoid argument, not because it's right. That could quietly build bad habits in schoolwork.
Sam: It does, highlighting a gap: research pushes tech fixes like adding fact-checks, but overlooks tuning student strategies to these quirks for better AI use.
Alex: That echo chamber effect from sycophancy is concerning. But going back, the study digs into how these seniors picture why hallucinations happen at all?
Sam: Yes, the paper groups their explanations into four main ways of thinking about it. One common view treats the AI like a search engine that digs through a stored collection of facts. When it hits a gap in that collection, students figure it just fills in with made-up stuff that sounds good. They call this the search-engine model. Another group sees it as spotting patterns from examples, like predicting the next move in a game by what happened before. They get that it picks likely words to keep things flowing, even if wrong. That's the inductive pattern detection model.
Alex: Like imagining a giant filing cabinet where missing files get invented on the spot. Pattern prediction over fact-checking—smooth sentences from stats, not truth. What about the deeper ones?
Sam: Some students point out the AI has no real grasp of meaning; it links words by chance without knowing if they're true, like reciting lines without understanding the story. This ungrounded cognition model sees hallucinations as built-in, since there's no way for it to spot its own gaps. A smaller set blames fuzzy instructions from users, where unclear questions leave the AI guessing from thin info.
Alex: Instructions as the weak link—can't always spell out everything perfectly. So these pictures shape how they handle errors?
Sam: The study suggests yes; the search-engine idea overlooks the generation process, so teaching the pattern side could shift that. It pushes for lessons on checking outputs rigorously, not just gut feel, since smooth wrongs fool intuition—what they term epistemic vigilance. But limitations note this is just senior engineering students from one school, self-reported views, not tested accuracy.
Alex: Self-reports from tech-savvy kids, yet gaps remain... that grounds it well.
Sam: The paper flags that directly. Limited to 63 seniors in computer engineering at one school, it can't speak for everyone; plus, open online answers might miss deeper thoughts you'd get from talks or tests. Self-reports also risk overclaiming skills without proof in real tasks, like spotting subtle issues in unfamiliar fields.
Alex: Right, so no check on if their spotting matches what actually happens. What does that mean for fixing things in classrooms?
Sam: It points to next steps: test if reported habits hold up in controlled trials across tasks. Interventions could teach the real pattern-guessing mechanics over database myths, plus step-by-step checks like pulling from notes or books. Domain tweaks might help too—say, humanities folks verifying citations differently than coders testing scripts—and track changes over time.
Alex: So shifting from gut to routines, tailored by field. That could cut the risks we talked about.
Sam: The paper concludes LLMs spit out smooth but shaky stuff by design, and with spotty student views plus intuition reliance, unchecked wrongs build bad habits. AI lessons need to drill the *why*—next-word patterns without truth gates—and hands-on verify skills to safely use them in school or work. It's a solid nudge toward practical safeguards.
Alex: Well said, Sam. That's our close look at how students see AI slip-ups and what it means for teaching smarter use. Thanks for joining us on ResearchPod.