SHANKS: SIMULTANEOUS HEARING AND THINKING FOR SPOKEN LANGUAGE MODELS

Traditional spoken language models (SLMs) process input in a turn-based manner, waiting for the user to finish speaking before beginning to think or act. This creates significant latency and prevents the model from interacting dynamically during the user's turn. The researchers ask: can an SLM be trained to "think while listening" to enable real-time, low-latency interactions like mid-turn interruptions and proactive tool usage?

To address this, the authors propose SHANKS (Simultaneous Hearing and Thinking with Chunked Input Speech). The framework segments incoming audio into fixed-duration chunks (e.g., 4 seconds). As each chunk is received, the model generates "unspoken" chain-of-thought reasoning tokens. This interleaved process allows the model to continuously update its internal state based on new audio input. The model is trained using a cross-entropy loss on sequences that alternate between speech input chunks and internal reasoning chunks, with special tokens designated for triggering interruptions or external API calls.

SHANKS demonstrates significant improvements in two key real-time interaction scenarios:

• Error Correction: In a tutoring scenario where users describe math solutions, SHANKS identifies and interrupts incorrect steps with 37.1% higher accuracy than a baseline model that interrupts without thinking.
• Tool-Augmented Dialogue: In task-oriented settings (e.g., travel planning), SHANKS can initiate and complete 56.9% of required API calls while the user is still speaking, effectively hiding the latency of tool execution behind the user's ongoing speech.

This work moves SLMs closer to human-like conversational fluency. By enabling models to reason and act in parallel with human speech, SHANKS reduces the "dead air" typically required for model processing, making voice-based AI assistants feel more responsive, proactive, and capable of handling complex, multi-step tasks in real-time.