PODBENCH: A COMPREHENSIVE BENCHMARK FOR INSTRUCTION-AWARE AUDIO-ORIENTED PODCAST SCRIPT GENERATION | Chenning Xu et al. | ResearchPod

Transcript: PODBENCH: A COMPREHENSIVE BENCHMARK FOR INSTRUCTION-AWARE AUDIO-ORIENTED PODCAST SCRIPT GENERATION

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a genuinely tricky problem in artificial intelligence: why do AI models struggle to write natural-sounding podcast scripts, even when they follow every instruction they're given?

Sam: So the question is — a computer can summarize a long article just fine, but ask it to turn that summary into a real conversation, and something goes wrong?

Alex: Exactly. The researchers behind this paper give that problem a name: the "instruction-substance gap." Think of it like a student who follows every formatting rule for a school report — correct font, right number of paragraphs, proper headings — but forgets to actually research the topic. The structure is there. The substance isn't.

Sam: That's a useful way to put it. So how did they actually study this?

Alex: They built a testing system they call "PodBench." It gives an AI model a long document — sometimes up to around 21,000 words, which is roughly the length of a short novel — and asks it to turn that into a podcast script. Then they measure how well it does.

Sam: And how do you even grade something like that? A podcast script isn't like a maths test with a right answer.

Alex: Right, so they use a two-part system. The first pass checks whether the script follows the basic rules — does it have the right number of speakers, does it stay on topic, is it formatted correctly? The second pass uses a separate AI to judge whether the conversation is actually interesting and informative. One check for structure, one check for quality.

Sam: So you could score well on structure and still produce something dull and shallow.

Alex: That's precisely what they found. The bigger, more powerful models do tend to perform better overall. But here's what's worth paying attention to: smaller, less powerful models — the kind that researchers can run without expensive computing infrastructure — can actually close that gap, if they're trained to think through the script step by step before writing it.

Sam: Wait — so the secret isn't just throwing more computing power at the problem? It's about forcing the model to plan first?

Alex: Yes. When a model is required to reason through the task explicitly — almost like drafting an outline before writing — it handles the complexity better. It's less likely to lose the thread of what the original document was actually saying.

Sam: But you mentioned this substance gap. Even with planning, are these models still producing scripts that are shallow?

Alex: That's the central concern the paper raises. A model can learn to mimic the rhythm and tone of a real conversation — the back-and-forth, the questions, the moments of reflection — without actually conveying the important ideas accurately. It follows the "how" of a podcast but misses the "what."

PODBENCH: A COMPREHENSIVE BENCHMARK FOR INSTRUCTION-AWARE AUDIO-ORIENTED PODCAST SCRIPT GENERATION

Chenning Xu, Mao Zheng, Mingyu Zheng, Mingyang Song

Jun 28, 2026·3:34·zotero·Original Paper

Podcast Script GenerationPodBenchInstruction-Aware GenerationEvaluation Framework

Listen

4 min

Briefing

Abstract

Podcast script generation requires LLMs to synthesize structured, context-grounded dialogue from diverse inputs, yet systematic evaluation resources for this task remain limited. To bridge this gap, we introduce PodBench, a benchmark comprising 800 samples with inputs up to 21K tokens and complex multi-speaker instructions. We propose a multifaceted evaluation framework that integrates quantitative constraints with LLM-based quality assessment. Extensive experiments reveal that while proprietary models generally excel, open-source models equipped with explicit reasoning demonstrate superior robustness in handling long contexts and multi-speaker coordination compared to standard baselines. However, our analysis uncovers a persistent divergence where high instruction following does not guarantee high content substance. PodBench offers a reproducible testbed to address these challenges in long-form, audio-centric generation.

Research Context and Objective

Podcast script generation is a specialized task for Large Language Models (LLMs) that requires synthesizing structured, multi-speaker dialogue from diverse, long-form inputs. Despite the growing popularity of AI-generated audio content, there has been a lack of systematic, reproducible benchmarks to evaluate how well models handle the specific constraints of podcasting, such as maintaining character consistency and managing long-context information. The authors introduce PodBench to address this gap.

The PodBench Benchmark

PodBench consists of 800 samples designed to test models on inputs reaching up to 21,000 tokens. The benchmark focuses on complex, multi-speaker instructions, requiring models to balance structural requirements with natural conversational flow. To evaluate performance, the authors propose a multifaceted framework that combines quantitative constraints—such as adherence to speaker roles and length requirements—with LLM-based quality assessments that judge the coherence and engagement of the generated scripts.

Key Findings

Experimental results indicate that proprietary models currently lead in overall performance. However, a significant finding is that open-source models equipped with explicit reasoning mechanisms demonstrate superior robustness when handling the challenges of long-context windows and multi-speaker coordination compared to standard baseline models. A critical takeaway from the study is the observed divergence between instruction following and content quality: models that excel at following structural instructions do not necessarily produce substantive or high-quality information, highlighting a need for better alignment between formatting and content generation.

Why It Matters

As LLMs are increasingly used to automate content creation for audio platforms, PodBench provides a necessary testbed for researchers to measure progress in long-form, audio-centric generation. By identifying the specific failure modes of current models—particularly the trade-off between structural compliance and content depth—this work guides future development toward more reliable and substantive AI-generated media.

Key Terms and Definitions

PodBench — A benchmark dataset containing 800 samples designed to evaluate the ability of LLMs to generate structured, multi-speaker podcast scripts from long-form inputs.
Multi-speaker Coordination — The ability of a model to maintain distinct, consistent personas and conversational turns across a script involving multiple participants.
Instruction-Aware Generation — A generation task where the model must strictly adhere to specific formatting or structural constraints provided in the input prompt.
Content Substance — A qualitative measure of the depth, accuracy, and meaningfulness of the information provided within a generated script, distinct from mere structural compliance.

PODBENCH: A COMPREHENSIVE BENCHMARK FOR INSTRUCTION-AWARE AUDIO-ORIENTED PODCAST SCRIPT GENERATION: Research Paper Podcast