Bagpiper-TTS: Natural Language Guided Universal Speech Synthesis

Traditional text-to-speech (TTS) systems are typically constrained by rigid input formats, requiring users to fill predefined metadata slots for speaker identity, emotion, or prosody. This approach limits flexibility, especially for complex, real-world scenarios. Bagpiper-TTS introduces a paradigm shift by using natural language as a universal interface, allowing the system to interpret free-form user requests directly.

Bagpiper-TTS operates through a three-stage hierarchical process executed within a single unified model:

1. Textual Planning: The model analyzes the user's intent to outline a conceptual summary, including transcription, speech events, and stylistic constraints.
2. Rich Caption Synthesis: Based on the plan, the model generates a dense textual blueprint—a rich caption—that formalizes all paralinguistic and acoustic requirements.
3. Speech Generation: The rich caption serves as the final guidance for the audio foundational model (Bagpiper-Base) to synthesize the target speech.

This architecture allows the system to handle diverse tasks, including multi-talker dialogue, intent-to-speech (where the model infers the transcription from context), role-play, and singing voice synthesis, all without requiring specialized modules for each task.

To enable this flexibility, the authors developed a sophisticated data simulation pipeline. They used large language models (LLMs) to reverse-engineer potential user requests from high-quality speech data, creating a large-scale training corpus of triplets: user requests, planning processes, and rich captions. The model was validated on the Seed-TTS-Eval benchmark, achieving a 1.7% Word Error Rate (WER) for classical TTS. In advanced applications, the model demonstrated strong performance, achieving a mean task fulfillment score of 4.09/5 in LLM-as-a-judge evaluations and 3.69/5 in human subjective testing.