The Significance of Style Diversity in Annotation-Free Synthetic Data Generation

In industrial settings where human-annotated data is scarce or expensive, how can we generate high-utility synthetic dialogue data for intent classification without relying on any human-annotated seed data? Specifically, the authors investigate whether controlling for topic and style attributes during generation can improve the performance of downstream intent classification models.

The researchers propose an annotation-free framework that generates synthetic dialogues using only intent definitions. The framework employs two primary strategies to enhance diversity:

1. Attribute-based Generation: The model is prompted with specific topic and style attributes to guide the generation process.
2. Post-hoc Stylization: Two novel models, 'Univ' (Universal) and 'Exam' (Example-based), are introduced to transform synthetic utterances into more human-like linguistic styles.

To ensure quality, the authors use an 'LLM-as-a-judge' filtering process, where a separate LLM validates whether the generated utterances correctly align with the intended intent. The framework was evaluated on both public (SGD) and proprietary industrial (EIC) datasets.

The study reveals that style diversity is a more critical factor than topic diversity for the utility of synthetic data. While topic diversity provides some benefit, style diversity is essential for preventing models from learning spurious correlations between stylistic patterns and specific intent classes. Furthermore, the authors find that incorporating style attributes directly during the generation process is more effective than attempting to adapt the style of generated text post-hoc. The proposed framework achieved up to 93.3% of the performance of models trained on human-annotated data.

This research provides a scalable solution for companies needing to deploy or update intent classification models in new domains or regions where no prior annotated data exists. By highlighting the importance of style over topic, the paper offers a clear directive for practitioners: when generating synthetic data, focusing on linguistic variation is more likely to yield robust, generalizable models than simply varying the subject matter.