VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis | Mengzhe Geng | ResearchPod

Transcript: VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis

Sam: A convincing voice can still be following the wrong evidence. VoxReason checks whether a speech system’s delivery choices follow the supplied record before any audio is generated.

Alex: So sounding appropriate is not enough. Does this work actually test the voice, or the decisions that produce it?

Sam: The decisions. This is a benchmark and method paper by Mengzhe Geng at the National Research Council Canada, available as an arXiv preprint. It makes those decisions explicit and checks their support.

Alex: That matters for dialogue agents and narration, where the same words can need very different delivery. What should researchers take from the result?

Sam: In this controlled setting, supplying the source evidence improved justified delivery choices much more than increasing model size. We’ll unpack how the check works, why ordinary accuracy can mislead, and where the evidence stops.

Alex: Start with the missing measurement. Context-aware speech systems already use information beyond the words. What does this add that evaluating their final audio cannot tell us?

Sam: The paper identifies a hidden decision layer before synthesis, the step that turns a plan into sound. Systems choose affect, pitch, energy, speed, pauses, emphasis, and stance. Once those choices are absorbed into audio, a plausible result does not reveal which source record justified them.

Alex: A system could recognize an emotion word and reproduce a familiar delivery pattern. That might sound reasonable without demonstrating that it followed the particular record.

Sam: That is the shortcut the paper targets. Existing evaluations often score final answers, audio, or rated outcomes. VoxReason instead asks for a speaking plan: explicit choices about delivery, with citations to the evidence supporting them.

Alex: Citations can also be decorative. A fluent explanation could name some context after the model has already chosen its answer. How does the benchmark make that harder?

Sam: It uses a deterministic verifier, a checker with fixed rules. The cited record must exist in that case. The delivery choice must agree with the label that record supports, and the plan must not invent speaker or scene information.

Alex: That catches unsupported answers. But what distinguishes actually following a cue from memorizing the answer associated with it?

Sam: The crucial test changes a cue while keeping the utterance fixed. The checker asks whether the choices linked to that cue change, while unrelated fields stay stable. The paper calls this counterfactual locality: a small evidence change should produce only the supported plan changes.

VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis

Mengzhe Geng

Sep 16, 2026·8:40·arxiv·Original Paper

Source-Grounded Speech PlanningVoxReasonListener-Free EvaluationCounterfactual LocalitySource-grounded speech planningListener-free verifier

Listen

9 min

Summary

Research Question and Approach

Expressive speech systems, such as dialogue agents and role-conditioned text-to-speech (TTS) models, must make critical delivery decisions—such as pitch, energy, rate, and stance—before generating audio. A significant evaluation gap exists: current metrics often focus on the final waveform, making it difficult to determine if a system’s delivery is genuinely licensed by the source context or if the system is simply producing plausible-sounding speech from a memorized script. VoxReason addresses this by casting speech planning as a pre-synthesis, source-grounded prediction task. Systems are required to output a speaking plan that includes explicit evidence citations, which are then verified by a deterministic system checking for citation legality, slot agreement, and counterfactual locality.

Main Findings

The study establishes that source-grounded planning is measurable and distinct from surface-level fluency. By using a controlled source-label testbed, the researchers show that when the same utterance is paired with different licensing cues, a robust planner must change its delivery fields accordingly. The results indicate that providing access to source records yields a substantially higher citation-required grounded score and plan-slot accuracy compared to transcript-only baselines. Crucially, the study finds that restoring the source-record channel within a 7B parameter model family produces a much larger performance gain than simply scaling the model from 3B to 7B parameters, suggesting that architectural access to evidence is more impactful than model size in this task.

Why It Matters

This work shifts the focus of speech evaluation from downstream audio quality to the underlying reasoning process. By requiring systems to justify their delivery choices with citations before synthesis, researchers can identify and mitigate failure modes where models rely on memorized social scripts rather than the specific context provided. This framework provides a diagnostic tool for developers to ensure that expressive speech systems are accountable to their source records, preventing the common issue of models sounding correct for the wrong reasons.

Key Terms and Definitions

Source-Grounded Speech Planning — The process of determining utterance delivery features, such as affect and prosody, based on explicit evidence from a source record.
Citation-Required Grounded Score — A metric that penalizes plans that fail to cite the specific source record required to justify the chosen delivery fields.
One-Cue Counterfactual Locality — A verification check ensuring that when a single source cue is changed, the system only updates the specific delivery slots linked to that cue, rather than rewriting the entire plan.
Shortcut Control — A baseline or test condition designed to reveal if a model is achieving high accuracy through memorization or priors rather than genuine evidence-based reasoning.
Listener-Free Evaluation — A method of assessing model performance using deterministic verifiers and structured outputs, avoiding the need for human listening studies or subjective audio ratings.

Abstract

Expressive speech systems make a decision before any waveform is rendered: how an utterance is delivered. In dialogue agents, narration, and role-conditioned TTS, that hidden planning step sets affect, pitch, energy, rate, pause, emphasis, and stance, yet downstream audio scores rarely reveal whether those choices were licensed by the source record, a source-use failure that occurs before any waveform exists. VoxReason makes that pre-synthesis decision measurable as a listener-free task for source-grounded speech planning. Before synthesis, VoxReason measures whether delivery choices are grounded in cited source records. Systems output a source-cited speaking-plan with evidence citations, and a deterministic verifier checks citation legality, slot agreement, unsupported state, schema validity, and one-cue counterfactual locality. On 1,440 checked source-label cases, shortcut controls show why slot accuracy alone is unsafe: a key-lookup oracle reaches 1.000 plan-slot accuracy on seen keys, while an emotion prior still reaches 0.958 slot accuracy on source-key-disjoint cases without citing intensity or identity. In a separate 100-case learned source-key-disjoint comparison, a 7B locality SFT+CF repair improves plan-slot accuracy/locality from 0.684/0.141 to 0.919/1.000, and removing source records lowers citation-required grounded score by 0.488. Rendered waveform quality remains outside the present evaluation.

VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis: Research Paper Podcast