Speech and audio generation is often needed in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production. In these scenarios, creators may need to design voices without reference recordings, control speaker styles with natural language, support acoustic scenes with environments and audio effects, and later reuse the designed voices. Therefore, it is important to support multi-speaker speech and audio generation for both instruct and zero-shot tasks. The instruct task requires a caption of the environment, speaker styles, and fine-grained content, while the zero-shot task uses reference audio together with the same fine-grained content. We address these tasks from both the data and model sides. First, we propose SwanData-Caption, which cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse and accurate multi-level captions. Then, we propose SwanTale, a multi-speaker expressive speech and audio generation model that supports both zero-shot and instruct tasks. We introduce SwanVAE to support high-quality multi-audio-modality generation. Then, we adopt reward-conditioned quality control and Engram conditioning, along with Unified MoE for multi-task and multi-audio-modality modeling. In addition, we use curriculum learning and GRPO post-training to let the model progressively learn and strengthen its capabilities. Experimental results show that SwanTale leads on multiple key zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and supports complex instruct generation involving multi-speaker speech and audio. Demos can be found at https://swanaigc.github.io/\#swantale.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a system called SwanTale, which tries to solve a specific problem in AI-generated audio: right now, if you want a computer to create a synthetic voice, you have to choose between two very different approaches.
Sam: What are the two approaches?
Alex: The first is called "zero-shot" voice cloning. You hand the system a short recording of someone's voice, and it copies that voice to say new things. The second is prompt-based generation, where you type a description—something like "a cheerful elderly woman with a soft accent"—and the system invents a voice that matches.
Sam: And the problem is that current systems are good at one or the other, but not both?
Alex: Right. Imagine you're creating an audiobook. You want to design a character's voice from scratch using a text description, but then you also need that same voice to stay consistent across dozens of chapters. Current tools make that surprisingly difficult. SwanTale is the researchers' attempt to handle both within a single system.
Sam: So how do they build something that can do two quite different things without getting tangled up?
Alex: Think about a large restaurant kitchen. There's a head chef who receives every order, but instead of cooking everything personally, they route each dish to the station best equipped to handle it—the grill station for steaks, the pastry station for desserts. SwanTale uses a similar idea. Rather than one single neural network trying to handle voice identity, background sounds, and emotional tone all at once, the system has a collection of specialised sub-networks. The researchers call this a "Unified Mixture-of-Experts." Each sub-network handles a narrower job.
Sam: So if you're generating a character speaking in a rainy alleyway, one expert handles the voice itself, and a different expert handles the rain in the background?
Alex: Exactly. And keeping those concerns separate matters, because if they compete for the same processing resources, you tend to get audio that's muddy in both dimensions—the voice doesn't sound quite right, and the environment doesn't either. Separating them lets each part do its job cleanly.
Sam: That makes sense. What about the raw audio itself—how does the system actually handle sound as data?
Alex: They built a component called SwanVAE. Audio files are enormous when stored in their raw form—far too large for the model to process efficiently. SwanVAE acts like a high-quality compression system. It translates raw audio into a much more compact representation that preserves the important details, works on it, and then translates back into audio you can actually hear. The key is that the compression is careful enough that nothing important gets lost in translation.
Sam: And I imagine the data side of this is just as important as the architecture. Where do you even find enough audio with detailed enough descriptions to train something like this?
Alex: That's genuinely one of the harder problems they faced. You can find large amounts of audio online, but almost none of it comes with the kind of rich, structured descriptions the model needs—things like the speaker's age, the acoustic quality of the room, what background sounds are present. So they built a pipeline called SwanData-Caption. It takes raw audio and feeds it through other AI models that automatically generate those detailed descriptions.
Sam: So they're essentially teaching the model to understand audio by writing it thorough notes first.
Alex: That's a good way to put it. They also identified a practical gap: certain kinds of voices and scenarios are genuinely rare in real-world recordings—elderly speakers, unusual proper nouns, very specific acoustic environments. For those cases, they generated synthetic examples deliberately, so the model wouldn't have blind spots when it encountered them.
Sam: It sounds like a substantial portion of the research effort was just making the training data good enough, before any model training even began.
Alex: That's accurate. Without structured, high-quality captions, the model has no reliable way to connect a text prompt like "a happy child playing in a park" to the specific combination of vocal qualities and ambient sounds that description implies. The data pipeline is what makes the text-to-audio link possible in the first place.
Sam: So once they had all of this built—the specialised experts, the compression system, the data pipeline—how did they test whether it worked?
Alex: They developed an evaluation framework called SwanBench. Rather than relying purely on human listeners, which is slow and expensive, they used a separate, highly capable AI model as an automated judge. That judge assessed the generated audio on several dimensions: how naturally the voice rose and fell in pitch and rhythm, how well the background sounds matched the description, and how expressive the overall result was.
Sam: Like having a very attentive listener grade each piece of audio against the original brief.
Alex: A reasonable analogy. Across the scenarios they tested—ranging from advertising copy to more complex multi-character scenes—SwanTale consistently scored highest on those dimensions compared to the other systems they evaluated.
Sam: But what does it still struggle with?
Alex: The authors were transparent about this. Long-form generation—anything running beyond roughly two minutes—remains difficult, particularly when multiple speakers and layered sound effects are involved simultaneously. Keeping track of that many moving parts over that duration is computationally demanding.
Sam: And I'd imagine emotional nuance is tricky too—like having a character's voice gradually shift from calm to distressed mid-sentence, rather than just being one or the other.
Alex: That's precisely the second limitation they flag. Precise, continuous emotional control within a single piece of audio is still a difficult problem. The system handles broad emotional tones reasonably well, but fine-grained, moment-to-moment shifts are not yet reliable.
Sam: So the natural next step would be adding editing tools—rather than regenerating an entire clip because one moment doesn't feel right, you'd be able to go in and adjust just that section.
Alex: That's exactly the direction the authors point toward. Unifying generation and editing within a single framework is the stated goal for future work. The underlying insight of SwanTale—that audio quality and acoustic environment should be controllable inputs, not just hoped-for outputs—is what would make that kind of editing possible. Thanks for listening to ResearchPod.