On the Role of Conversational Timing in Synthetic Training Data for ASR

This paper investigates how the timing of conversational interactions—specifically the duration of pauses and the frequency of overlapping speech—affects the performance of Automatic Speech Recognition (ASR) systems trained on synthetic data. Rather than treating conversational timing as a static property to be replicated from a target corpus, the authors parameterize timing distributions using an exponential-tilting family. This allows them to treat timing as a controllable variable, enabling systematic exploration of the parameter space using Latin hypercube sampling and multi-objective Bayesian optimization.

The researchers found that downstream ASR performance is highly sensitive to the specific timing statistics induced during data simulation. Notably, increasing the exposure to overlapping speech was associated with lower concatenated-permutation word error rates (cpWER). Conversely, longer and more variable silence gaps between utterances were linked to higher error rates. The Bayesian optimization process revealed an inherent overlap-gap trade-off, suggesting that the most effective training data are not necessarily those that perfectly mirror the statistics of a single source corpus, but rather those that provide a specific, task-relevant profile of timing variability.

Most current speech synthesis pipelines prioritize realism, aiming to match the timing distributions of real-world conversational corpora. This study challenges that paradigm by suggesting that "realistic" timing is not always "optimal" for training. By providing a framework to manipulate and optimize timing parameters, the authors offer a diagnostic tool for researchers to understand how specific interaction dynamics impact model robustness. This shift from descriptive simulation to prescriptive, performance-oriented simulation is a significant step toward building more reliable conversational AI.