Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

As Spoken Language Models (SLMs) shift from rigid, turn-based half-duplex interaction to fluid, full-duplex communication, they face a significant performance bottleneck: modality interference. When acoustic and semantic modeling are forced to share a deep parameter space, the model suffers from knowledge degradation and compromised semantic integrity. This paper investigates the root cause of this interference and proposes a novel architectural solution to achieve high-performance, native full-duplex interaction.

Through a fine-grained analysis of model optimization dynamics, the authors quantify the geometric relationship between gradient vectors of semantic and acoustic objectives. They reveal that while shallow layers exhibit task synergy, deep layers suffer from severe gradient conflicts and semantic dilution—where sparse text tokens are overwhelmed by dense audio frames. To address this, they introduce Lychee-FD, a framework featuring two key innovations:

1. Hierarchical Parameter Separation: The model uses a shared backbone for shallow layers but splits into specialized semantic, acoustic, and control heads in deeper layers to isolate conflicting optimization trajectories.
2. Semantic Alignment Channel: A dedicated channel provides continuous textual supervision, acting as a semantic anchor to prevent the dilution of language modeling objectives.

Lychee-FD significantly outperforms existing full-duplex SLMs. It achieves a 7.4% improvement in spoken question answering accuracy and a 28.5% gain in interaction fluidity on the FullDuplexBench 1.5 benchmark. By utilizing a Directed Acyclic Graph Pipeline Parallelism (DAG-PP) during inference, the model maintains the efficiency of a native end-to-end architecture while resolving the modality conflict. The authors demonstrate that their approach not only recovers the performance of the half-duplex backbone but surpasses it, confirming that explicit semantic modeling is essential for high-intelligence, real-time voice interaction.