Measuring Users’ Mental Models of Speech Translation in Human-AI Collaboration

This paper investigates how users form and refine their mental models (MMs) of machine translation (MT) systems. While MT is used daily, users often lack a clear understanding of when these systems succeed or fail. The authors propose a new framework based on cross-lingual question answering (QA). In this setup, users are presented with French audio and its machine-translated English text, then tasked with answering comprehension questions. To maximize their performance, users must decide whether to accept the MT output as-is or pay a penalty to request a professional re-translation of specific segments. This task forces users to perform a cost-benefit analysis, which the researchers use to measure how well users identify MT error patterns over time.

The study finds that users consistently improve their mental models as they gain experience with the task, evidenced by increasing rewards and decreasing "simple regret" (the gap between their actual performance and the optimal strategy). Language proficiency plays a significant role: while fluent and intermediate users show clear improvements in their ability to predict system failures, basic users struggle to update their mental models effectively.

Regarding error detection, users are most successful at identifying surface-level errors, such as incomplete or unnatural-sounding translations, while topic-specific errors remain difficult to spot. The researchers also tested different forms of feedback. Providing speech transcriptions proved highly effective, as it offered users additional clues about the source audio. Conversely, highlighting specific error spans improved immediate accuracy but discouraged deeper learning, as users became overly reliant on the system's automated cues rather than developing their own intuition.

Understanding user mental models is critical for effective human-AI collaboration. By framing MT quality as "fitness for purpose" rather than just a static metric, this research provides a practical way to measure and improve MT literacy. The findings suggest that providing users with supplementary information, like transcriptions, can significantly help them navigate the limitations of speech translation in high-stakes environments.