Transcript: Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition

Alex: Welcome to another episode of ResearchPod. Sam, what paper caught your eye this time?

Sam: This paper, titled "Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition," looks at turning speech into text across multiple languages using a two-step process. The key puzzle it tackles is how to build one shared model that handles text spelling for ten different languages at once, matching or beating the old methods that need separate setups for each language.

Alex: So this is basically about making speech recognition work smoothly for places like call centers that get calls in English, Spanish, French, and even less common ones like Kyrgyz or Tatar—all mixed together?

Sam: Yes, exactly. Traditional systems break speech recognition into sounds first, then spelling those sounds into words. But the usual way to do the spelling part relies on custom maps and graphs built just for one language each time, which gets bulky and hard to manage when you're dealing with ten or more.

Alex: Right, so if a call comes in a mix of languages, those old systems would need to guess the language upfront or build one giant graph that explodes in size.

Sam: That's the core problem. They call the sound-breaking step speech-to-phoneme—the first stage listens to audio and guesses the basic sound units, like the building blocks of pronunciation that many languages share. The second step, phoneme-to-grapheme, takes those sound units and turns them into actual letters or words, but each language spells the same sounds differently, like how "phone" in English uses "ph" but might look different in French or Russian.

Alex: Okay, so sharing the sound-guessing part across languages makes sense because sounds overlap a lot. But the spelling part has to know exactly which language's rules to follow, without separate machines for each.

Sam: Precisely. The paper uses a dataset called CV-Lang10, which covers ten languages from high-data ones like English and Italian down to low-data ones like Tatar. Old spelling systems, known as WFSTs, work well but demand a full dictionary and graph per language, plus they assume you already know what language you're hearing—which doesn't hold in mixed calls.

Alex: And these large language models they're mentioning—aren't those the same ones that chat and write stories?

Sam: Yes, the researchers fine-tune a large language model for that spelling step. It learns to spot the language from the sounds and output the right spelling, all in one model. But challenges pop up: the model has to generate text aware of each language's quirks, and training data is uneven—some languages have thousands of hours, others just tens, so the model ignores the rare ones.