Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah, Ricardo Gutierrez-Osuna
8 min
Abstract
Speaker anonymization (SA) systems modify timbre while leaving regional or non-native accents intact, which is problematic because accents can narrow the anonymity set. To address this issue, we present PHONOS, a streaming module for real-time SA that neutralizes non-native accent to sound native-like. Our approach pre-generates golden speaker utterances that preserve source timbre and rhythm but replace foreign segmentals with native ones using silence-aware DTW alignment and zero-shot voice conversion. These utterances supervise a causal accent translator that maps non-native content tokens to native equivalents with at most 40ms look-ahead, trained using joint cross-entropy and CTC losses. Our evaluations show an 81% reduction in non-native accent confidence, with listening-test ratings consistent with this shift, and reduced speaker linkability as accent-neutralized utterances move away from the original speaker in embedding space while having latency under 241 ms on single GPU.
Alex: So it's like dubbing a movie: swap the words' sounds to perfect English, but keep the actor's timing and voice color so it feels natural.
Sam: Precisely. These golden waveforms become the training targets. In stage two, they train the accent translator—a model that takes the non-native speech, breaks it into content pieces called tokens, which are like phonetic building blocks independent of the voice, and swaps the non-native tokens for native ones.
Alex: Right, the token swap sounds key. But inside that accent translator, how does it actually figure out what native tokens to pick for the non-native ones?
Sam: It processes the incoming tokens step by step to catch patterns in how sounds connect—like scanning a few letters at a time in a word to spot common blends. First, a set of layers handles short-range links, only peeking at what's already come before to keep things real-time. Then a middle section links sounds across words, using recent history plus a tiny 40-millisecond glimpse ahead. Finally, more close-up layers refine it before picking from possible tokens, including a blank for pauses.
Alex: Okay, so it's layering local fixes with some broader context, but staying causal to avoid delays. What trains it to make those swaps accurate?
Sam: They use two guides during training to compare outputs to the golden speaker targets. One matches sounds frame by frame. The other overlooks small timing mismatches by just checking if the overall sequence of tokens matches the native one. This combo makes it robust: precise where possible, flexible otherwise.
Alex: And for live use, does it just pick the top match every time, or add some variety?
Sam: In real-time, it runs fully online, handling speech chunk by chunk through encoder, translator, and decoder, swapping tokens before rebuilding the waveform. To avoid robotic repetition, it samples from the top likely tokens, adding natural variety like slight word flavor differences in speech.
Alex: They trained on Indian-accented data, I assume? How'd they test if it actually fools accent detectors?
Sam: Training used 250 hours from IndicTTS—Indian speakers reading English—with golden pairs made from native text-to-speech on the same scripts. Evaluation came from L2-ARCTIC's Indian subset: 100 utterances each from four speakers. An off-the-shelf classifier judged native-likeness. PHONOS hit 97 percent native classification, close to the golden speaker upper bound—a clear improvement over plain Indian baselines.
Alex: Huh. So near-perfect neutralization in tests. But what about overall sound quality or if it still sounds like the original speaker?
Sam: The paper checks synthesis quality separately and finds PHONOS preserves naturalness well, comparable to reconstruction baselines. On speaker verifiability, it maintains recognition rates near originals, meaning the core identity holds despite accent shifts. This suggests meaningful privacy gains without breaking voice usability in calls.
Alex: So quality holds up, and the speaker still seems recognizable. But how exactly did they measure if the accent really vanished?
Sam: They fed the processed speech into an off-the-shelf accent classifier, trained to spot native versus non-native patterns. It labels each utterance as native or not, and tracks confidence shifts. PHONOS reached 97 percent classified as native—very close to the offline golden speaker's 99 percent—while plain reconstruction only hit 57 percent. Listeners rated perceived accentedness lower too.
Alex: Interesting—the classifier shift shows it's not just partial cleanup from the base system. What explains that extra push to near-native?
Sam: The base reconstruction partially normalizes because it squeezes non-native sounds into a shared set of phonetic building blocks trained mostly on native speech. PHONOS's translator adds a dedicated step to swap those for cleaner native matches, closing most of the remaining gap. The paper notes the slight shortfall likely comes from real-time limits, like only peeking 40 milliseconds ahead.
Alex: Got it—so the streaming setup trades a tiny bit of perfection for speed. Speaking of which, how fast does the whole thing actually run in practice?
Sam: On a standard graphics card like an Nvidia RTX 3090, it processes speech chunks with a total delay of 241 milliseconds end-to-end. Even on a regular computer processor, it reaches real-time speeds at 370 milliseconds total. This stays low enough for natural video calls.
Alex: Huh. Low delay without quality hits. Any catches they mention?
Sam: Yes, the tests focused on just four Indian English speakers, so broader accents or more people need checking. It also skips rhythm tweaks, leaving some accent hints. Overall, the paper positions this as a useful add-on to voice tools, cutting linkability from accent cues while keeping speech clear.
Alex: Makes sense—a targeted fix that builds privacy without overhauling everything. That's a grounded contribution—fixing an overlooked gap thoughtfully. Thanks for breaking it down, Sam. And that's our look at PHONOS from ResearchPod.