ResearchPod Summary
As voice agents become standard for customer-facing applications, existing benchmarks primarily focus on component-level metrics like word error rate or latency. However, these metrics fail to capture whether an agent can successfully resolve a customer's request—a metric contact centers call "containment." VAmoS Bench was developed to address this gap by evaluating complete, end-to-end voice agent systems on their ability to handle stateful, multi-turn tasks without human intervention.
The authors introduce a simulation-based platform where each voice agent interacts with a simulated caller over live audio. The benchmark consists of 100 scenarios centered on a credit-card support task for a fictional bank. Each scenario provides an isolated, seeded PostgreSQL database and a specific caller goal. The agent must navigate these goals using five available tools to perform actions like freezing or replacing a card.
A critical feature of VAmoS is its joint grading system. Rather than evaluating speech and database state separately, the grader analyzes the complete trace of the conversation, tool invocations, and database changes. This allows the benchmark to detect failures where an agent might claim to have performed an action without actually executing it, or where an agent discloses protected information before properly verifying the caller's identity.
The authors evaluated eleven different voice agent stacks, including self-hosted frameworks, hosted platforms, and native speech-to-speech models. The results show that task completion rates vary significantly, ranging from 43% to 71% across the tested systems. Notably, the vast majority of phone calls (98.3%) successfully connected, indicating that the primary failure points occur during the reasoning and execution phases of the interaction rather than at the infrastructure or connectivity level.
By moving beyond component-level metrics, VAmoS Bench provides a more realistic assessment of how voice agents perform in production-like environments. It forces developers to consider the entire pipeline—from audio processing to tool-use logic—and highlights the importance of consistency between what an agent says and what it actually does in the backend. This framework provides a standardized way to compare diverse architectural approaches, such as cascaded versus native speech-to-speech models, under identical operational conditions.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.