Bhavana Akkiraju, Ravi Sastry Kolluru, Charan Devarakonda, Srihari Bandarupalli, Santosh Kesiraju, Anil Kumar Vuppala
5 min
Question answering (QA) research has largely focused on high-resource languages, leaving low-resource languages like Telugu under-explored, particularly in spoken settings. The authors address this gap by introducing VākQA, a benchmark consisting of 2,001 factoid question-answer pairs across six domains, including 2.53 hours of human-verified spoken Telugu audio, transcriptions, and English translations. The researchers evaluate both proprietary models (Gemini) and various open-weight models (e.g., Gemma-3, Llama-3.1) across different input modalities (speech vs. text) and languages (Telugu vs. English) to understand how model performance is impacted by input conditions and cascaded error propagation.
Central to this work is the assessment of automatic evaluation methods. The authors compare human judgments against lexical metrics (Exact Match, F1), embedding-based metrics (BLASER-2.0), and LLM-as-a-judge approaches. They find that traditional lexical metrics are brittle and fail to capture semantic equivalence in Telugu. Among LLM judges, Gemini-as-a-judge shows the highest correlation with human ratings, though it exhibits non-uniform strictness—being more lenient with low-quality answers and stricter with high-quality ones. Open-weight models often struggle to recognize semantic equivalence, leading to systematic penalization of correct answers that differ in surface form from the reference.
This study highlights the limitations of current evaluation practices for low-resource languages and provides a robust benchmark to guide future development of spoken QA systems. It demonstrates that simply translating benchmarks is insufficient, as cultural and linguistic nuances are often lost, and emphasizes the need for more reliable, language-aware evaluation metrics.
Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. We first validate evaluation methods against human judgements: Gemini-as-a-judge best approximates human ratings but is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference. Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively. VākQA is publicly released.
Alex: [analytical] They did, and they tested several options. Gemini-as-a-judge best approximated human ratings, but it wasn't uniformly well-calibrated. It was non-uniformly strict — meaning it penalized answers where the surface form diverged from the reference even when the semantic content was correct. That's a real problem for evaluation validity.
Sam: [reflecting] If the judge is systematically biased against paraphrase, you can't cleanly separate model failure from evaluator failure. How do they handle that? [[RP_SECTION:pipeline-ablation-analysis|Pipeline Ablation Analysis]]
Alex: [slower, for clarity] They don't fully resolve it, but they do something useful: they quantify the contribution of each pipeline stage by running models under different input conditions. Direct speech input, transcribed text, oracle text — comparing performance across those modalities lets them isolate how much degradation originates at ASR, how much at translation, and how much is residual model failure. It's essentially an ablation over the pipeline itself.
Sam: [nodding] So they're stress-testing the full stack, not just the final model.
Alex: [calm] Exactly. And the key result from that comparison is that the performance gap between oracle-text conditions and cascaded-pipeline conditions is large enough that it can't be attributed to model capacity. You could swap in a substantially better LLM and still hit the same floor, because the bottleneck is upstream. That's the load-bearing finding.
Sam: [analytical] Which has a fairly direct implication for how you'd prioritize engineering effort. [[RP_SECTION:engineering-implications|Engineering Implications]]
Alex: [measured] It does. If you're building a Telugu voice QA system and you're chasing benchmark numbers by scaling the language model, this paper suggests you're optimizing the wrong component. The marginal return on ASR and MT quality is higher than the marginal return on LLM size, at least until the pipeline errors are brought under control.
Sam: [reflective] And that's probably true for other low-resource languages with similar phonetic complexity — Telugu isn't unique in having this problem.
Alex: [considered] The paper doesn't generalize beyond Telugu, and it would be overreading to claim the finding transfers directly. But the mechanism — cascaded error propagation producing a semantic floor that model scaling can't overcome — is not specific to Telugu phonology. It's a structural property of the pipeline architecture. Whether the magnitude holds for other languages is an open question, and one the benchmark is well-positioned to help answer as it gets adopted.
Sam: [grounded] So the benchmark itself is the contribution as much as any specific result.
Alex: [concluding] That's a fair reading. The diagnostic value — being able to decompose pipeline failure across modalities — is arguably more durable than any single accuracy number. The numbers will shift as models improve. The framework for understanding where the errors originate is what this work actually contributes. Thanks for listening to ResearchPod.