Yu Zhang, Ruiqi Li, Changhao Pan, Ke Lei, Xiang Yin, Cheng Yang
6 min
Abstract
Speech and audio generation is often needed in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production. In these scenarios, creators may need to design voices without reference recordings, control speaker styles with natural language, support acoustic scenes with environments and audio effects, and later reuse the designed voices. Therefore, it is important to support multi-speaker speech and audio generation for both instruct and zero-shot tasks. The instruct task requires a caption of the environment, speaker styles, and fine-grained content, while the zero-shot task uses reference audio together with the same fine-grained content. We address these tasks from both the data and model sides. First, we propose SwanData-Caption, which cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse and accurate multi-level captions. Then, we propose SwanTale, a multi-speaker expressive speech and audio generation model that supports both zero-shot and instruct tasks. We introduce SwanVAE to support high-quality multi-audio-modality generation. Then, we adopt reward-conditioned quality control and Engram conditioning, along with Unified MoE for multi-task and multi-audio-modality modeling. In addition, we use curriculum learning and GRPO post-training to let the model progressively learn and strengthen its capabilities. Experimental results show that SwanTale leads on multiple key zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and supports complex instruct generation involving multi-speaker speech and audio. Demos can be found at https://swanaigc.github.io/\#swantale.
Sam: And I imagine the data side of this is just as important as the architecture. Where do you even find enough audio with detailed enough descriptions to train something like this?
Alex: That's genuinely one of the harder problems they faced. You can find large amounts of audio online, but almost none of it comes with the kind of rich, structured descriptions the model needs—things like the speaker's age, the acoustic quality of the room, what background sounds are present. So they built a pipeline called SwanData-Caption. It takes raw audio and feeds it through other AI models that automatically generate those detailed descriptions.
Sam: So they're essentially teaching the model to understand audio by writing it thorough notes first.
Alex: That's a good way to put it. They also identified a practical gap: certain kinds of voices and scenarios are genuinely rare in real-world recordings—elderly speakers, unusual proper nouns, very specific acoustic environments. For those cases, they generated synthetic examples deliberately, so the model wouldn't have blind spots when it encountered them.
Sam: It sounds like a substantial portion of the research effort was just making the training data good enough, before any model training even began.
Alex: That's accurate. Without structured, high-quality captions, the model has no reliable way to connect a text prompt like "a happy child playing in a park" to the specific combination of vocal qualities and ambient sounds that description implies. The data pipeline is what makes the text-to-audio link possible in the first place.
Sam: So once they had all of this built—the specialised experts, the compression system, the data pipeline—how did they test whether it worked?
Alex: They developed an evaluation framework called SwanBench. Rather than relying purely on human listeners, which is slow and expensive, they used a separate, highly capable AI model as an automated judge. That judge assessed the generated audio on several dimensions: how naturally the voice rose and fell in pitch and rhythm, how well the background sounds matched the description, and how expressive the overall result was.
Sam: Like having a very attentive listener grade each piece of audio against the original brief.
Alex: A reasonable analogy. Across the scenarios they tested—ranging from advertising copy to more complex multi-character scenes—SwanTale consistently scored highest on those dimensions compared to the other systems they evaluated.
Sam: But what does it still struggle with?
Alex: The authors were transparent about this. Long-form generation—anything running beyond roughly two minutes—remains difficult, particularly when multiple speakers and layered sound effects are involved simultaneously. Keeping track of that many moving parts over that duration is computationally demanding.
Sam: And I'd imagine emotional nuance is tricky too—like having a character's voice gradually shift from calm to distressed mid-sentence, rather than just being one or the other.
Alex: That's precisely the second limitation they flag. Precise, continuous emotional control within a single piece of audio is still a difficult problem. The system handles broad emotional tones reasonably well, but fine-grained, moment-to-moment shifts are not yet reliable.
Sam: So the natural next step would be adding editing tools—rather than regenerating an entire clip because one moment doesn't feel right, you'd be able to go in and adjust just that section.
Alex: That's exactly the direction the authors point toward. Unifying generation and editing within a single framework is the stated goal for future work. The underlying insight of SwanTale—that audio quality and acoustic environment should be controllable inputs, not just hoped-for outputs—is what would make that kind of editing possible. Thanks for listening to ResearchPod.