Zixuan Wang, Yuhong Chen, Yuxuan Zhu, Guidong Lei, Zhiluohan Guo, Yu Zhao, Kun Wang, Bangyang Hong, Kangle Wu, Yabo Ni, Anxiang Zeng, Cong Fu, Hui Li
6 min
Abstract
Industrial recommenders increasingly adopt the pretrain-then-transfer paradigm, yet behavioral distribution drift raises two questions: what to learn from behavior sequences, and how to transfer the learned knowledge while the pretrained model is continually refreshed. To resolve them, we propose Knowledge-Geometry Decoupling (KGD). For what to learn, conventional next-token prediction treats adjacency as dependency and may encode spurious transitions across unrelated sessions. We introduce Behavioral Multi-Token Prediction (BMTP) to retain only collaboratively or semantically related future items as supervision, yielding cleaner and more transferable behavioral knowledge. For how to transfer, pretrained knowledge and task-specific geometry impose conflicting optimization demands on shared parameters. To handle it, KGD assigns them to separate parameter sets: a refreshable encoder owns behavioral knowledge, while a task learner reads contextualized encoder states through read-only cross-attention and writes task-specific geometry through Anchored Calibration Residual (ACR) orthogonal to the pretrained embedding. The decoupled ownership enables continual knowledge refresh without task-gradient interference or invalidating downstream adaptation. KGD improves over strong pretrain-transfer baselines by 4-12% on eight public benchmarks and sustains its advantage over a 90-day production stream where baselines show no gains. KGD has been fully deployed in Shopee. In a live A/B test on Shopee Homepage Search, it increases GMV per user by 1.75% and advertising revenue by 1.53%, demonstrating its high practical value. We provide the core implementation of KGD at https://github.com/FuCongResearchSquad/KGD4REC.
Alex: You also mentioned "denoising" earlier. How does the system know what's genuinely useful versus just noise?
Sam: That's handled by something the paper calls "Behavioral Multi-Token Prediction." Here's the problem it's solving: standard training assumes that every item a user clicks is meaningfully connected to the next one. But real shopping behavior is messy. You might buy a bike part, then immediately buy a pair of trousers. Those two actions aren't really related—you just needed both things that day.
Alex: Oh—so the model was learning false patterns because it assumed everything in a session was part of one coherent story?
Sam: Exactly. If the model treats every click as a meaningful signal, it ends up learning a lot of noise alongside the genuine patterns. By filtering out transitions between items that aren't truly related, the system only trains on the connections that actually mean something. It's a bit like watching a skilled teacher—you'd learn more by focusing on the moments when they're actually explaining a concept, rather than treating every single thing they do in the room as a lesson.
Alex: And when they tested this in practice?
Sam: The method outperformed existing approaches by a meaningful margin in controlled tests. More tellingly, in a real-world deployment at Shopee—a major e-commerce platform—it produced a measurable lift in both sales and ad revenue. That's a useful signal, because lab results and live results don't always agree. Here they did.
Alex: That's worth noting. But this sounds like it adds complexity. Is there a cost to running two separate layers instead of one?
Sam: There is. Because you're maintaining two distinct components—the foundational encoder and the task-specific overlay—the system carries roughly double the memory of a standard model. The daily training time also increases, from around one hour to two. Those are real infrastructure costs that any team adopting this would need to plan for.
Alex: So the trade-off is: more compute and memory upfront, but a system that actually holds up over time instead of quietly degrading.
Sam: That's the argument the paper makes. The data they present tracks performance over a 90-day window. Other methods might look competitive in the short term, but they erode as time passes. KGD maintains its performance because it can keep refreshing the overlay without ever corrupting the foundation.
Alex: Looking ahead, does this point toward a world where one large model could serve many different tasks simultaneously, just by swapping out its overlay?
Sam: That's the direction this architecture suggests. The paper points toward what researchers describe as a "Universal Recommender Backbone"—a single foundation that could serve many different downstream tasks at once, each with its own overlay being refreshed independently, without any of them interfering with the others. It would remove the need to retrain from scratch every time a new business goal emerges.
Alex: So the insight isn't just about having more data—it's about being more careful with how you use and preserve what you already know.
Sam: That's a good way to put it. The study suggests that how you manage your knowledge is just as important as how much of it you have. By keeping the geometry separate from the foundation, you stop the system from fighting itself every time it learns something new.
Alex: It's a meaningful step toward making these systems more stable in practice. Thanks for walking me through this, Sam.
Sam: Happy to. The open question for the field is probably data efficiency—whether you can get these benefits without the added infrastructure cost. But as a solution to catastrophic forgetting in live recommendation systems, this is a significant step forward. Thanks for listening to ResearchPod.