Yuqi Li, Yi-Cheng Lin, Xianglong Wang, Kuo Yang, Xiaoqin Feng, Yixuan Wang, Huiran Duan, Yingli Tian
6 min
On-device speech emotion recognition (SER) requires compact models, but compressing high-performing self-supervised learning (SSL) models often results in significant performance loss. While multi-teacher knowledge distillation (MTKD) can mitigate this by aggregating diverse teacher knowledge, existing methods struggle with two issues: teacher reliability fluctuates across data batches, and standard logit-level distillation fails to capture the relational structure of emotions in the feature space. The authors propose Adaptive Multi-teacher Relational Distillation (AMRD) to address these limitations.
AMRD introduces two primary mechanisms. First, it uses a one-class SVM to analyze the logit similarity matrix of each teacher per batch, assigning higher weights to teachers that produce more coherent predictions. Second, it implements Relational Similarity Matrix Distillation (RSMD), which forces the student to mimic the pairwise cosine similarity patterns of the teacher's features. Both components are active only during training, ensuring no additional latency during inference.
The authors evaluated AMRD on the IEMOCAP and CREMA-D datasets using four student architectures ranging from 0.78M to 11.7M parameters. AMRD consistently outperformed single-teacher distillation baselines in seven out of eight tested student-dataset configurations. On the IEMOCAP dataset, the framework achieved up to a 2.9% improvement in unweighted accuracy compared to the best single-teacher baseline. Ablation studies confirmed that both the SVM-based dynamic weighting and the relational distillation loss contribute complementary gains to the final model performance.
This work demonstrates that effective knowledge transfer in SER requires more than just mimicking soft labels; it necessitates adapting to the varying reliability of teachers and preserving the structural relationships between emotional samples. By utilizing lightweight student models, AMRD enables high-quality emotion recognition on edge devices, making it a practical solution for real-time applications like mental health monitoring and human-computer interaction.
On-device speech emotion recognition (SER) is critical for real-time applications, yet large self-supervised models that excel at SER are too costly for edge devices. Multi-teacher knowledge distillation can compress them into a lightweight student, but two challenges remain: teacher reliability varies across batches, and logit-level distillation ignores inter-sample relational structure. We propose Adaptive Multi-teacher Relational Distillation (AMRD) to address both. A one-class SVM on each teacher's logit similarity matrix assigns per-batch weights favoring more coherent teachers. A relational distillation loss aligns teacher and student similarity matrices, capturing structure that logit matching misses. On IEMOCAP and CREMA-D datasets across four student architectures, AMRD outperforms single-teacher distillation baselines in most settings, and ablations confirm both components yield complementary gains.
Sam: So the student is always learning from whoever is most reliable at that particular moment.
Alex: That's the idea. And it matters because emotion in speech is genuinely subjective. The same phrase spoken by two different people can carry completely different emotional weight, so even a well-trained teacher model will have moments of uncertainty. The adaptive weighting means the student doesn't inherit those moments of confusion.
Sam: You mentioned "relational" distillation as well. What does that layer add?
Alex: This is the part I find most compelling. Standard distillation methods have the student look at each audio clip individually and try to match the teacher's prediction for that clip. But that ignores something important: the relationships between clips.
Sam: Can you give me a sense of what that means in practice?
Alex: Sure. Imagine you're learning to sort music by mood. If someone just tells you "this song is sad" and "that song is happy," you can memorize those labels. But you'd learn much more if you also understood that these two sad songs are similar to each other, while that third sad song has a completely different quality—more resigned than grief-stricken. The relationships carry information that the labels alone don't.
Sam: So the student is learning the shape of the emotional landscape, not just individual data points on it.
Alex: Precisely. The paper calls this Relational Similarity Matrix Distillation. For every batch of audio clips, the system builds a kind of map showing how all the clips relate to each other in emotional space. The student is then trained to reproduce that map, not just the individual predictions. It captures the structure of emotion, which is much harder to fake with a small model that's only seen isolated examples.
Sam: And once training is done, all that complexity disappears?
Alex: Yes, and that's a key practical point. All of this—the adaptive weighting, the relational maps, the multiple teachers—only exists during the training phase. Once the student model has internalized those lessons, it runs entirely on its own. You could deploy it on a phone or a small home device, and it would just work, with no need for the teachers or any of the training machinery.
Sam: So what did the results actually show? Did this approach outperform simpler methods?
Alex: The paper reports meaningful improvements. Their smallest student model is more than a hundred times smaller than the teacher models it learned from, and yet it achieved notably better accuracy than models trained using standard distillation methods. The gap wasn't marginal—it was a clear, consistent improvement across their tests.
Sam: Were the researchers candid about where the approach has limits?
Alex: They were. A few things worth noting. First, the current framework only uses two teacher models. Scaling up to more teachers would add significant training complexity, and they haven't fully worked that out yet. Second, the system is audio-only—it doesn't incorporate visual information like facial expressions or body language, which humans rely on heavily when reading emotions. So there's a real ceiling on what audio alone can tell you.
Sam: So it's a solid step forward for voice-based emotion recognition, but not the complete picture.
Alex: That's a fair summary. What the paper demonstrates, though, is a meaningful principle: how you teach a model matters as much as what you teach it. By being selective about which teacher to trust, and by teaching relationships rather than just isolated labels, you can train a genuinely capable small model—one that's practical enough to run on the devices people actually use. That's a worthwhile problem to have made progress on.
Sam: Thanks for walking us through it, Alex.
Alex: Thanks for listening to ResearchPod.