Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams

This study investigates the efficacy of large language models (LLMs) in detecting phone scams within the Turkish language, a low-resource setting often overlooked in AI safety research. By introducing a new multi-modal dataset of 100 aligned audio-transcript pairs, the authors evaluate how different input modalities—raw audio, uncorrected transcripts, and human-refined transcripts—impact the detection capabilities of seven prominent LLMs, including Gemini 2.5, GPT-4o, and Qwen.

The researchers tested three input methods: raw audio processing, automatic speech-to-text (ASR) transcripts, and transcripts manually corrected by native speakers. The goal was to determine if the prosodic and affective cues present in audio provide additional value for identifying fraudulent intent, or if the textual content is sufficient. The models were evaluated without fine-tuning to reflect standard, out-of-the-box performance.

The results demonstrate that transcript-based inputs consistently outperform raw audio. While audio processing is theoretically capable of capturing tone and stress, the models frequently triggered internal safety filters when encountering the aggressive language, profanity, or intimidation tactics common in scam calls. These false negatives significantly reduced the performance of audio-based detection. Interestingly, the study found that human-corrected transcripts provided negligible gains over raw ASR outputs, suggesting that current ASR systems are sufficiently accurate for this task and that manual intervention is not a cost-effective strategy for improving detection accuracy.

This work highlights a critical tension between AI safety mechanisms and practical utility. By prioritizing safety filters that block aggressive content, models may inadvertently become less effective at identifying the very threats they are designed to mitigate. Furthermore, the study underscores the need for more inclusive AI research that addresses the unique linguistic and cultural challenges of low-resource languages like Turkish.