TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems

Modern recommender systems often rely on dense user embeddings, which can struggle to capture fine-grained, long-term sequential interests. While Large Language Models (LLMs) offer an alternative for user representation, they often suffer from a modality gap when applied to non-textual item spaces. TokenMinds investigates whether combining discrete Semantic ID (SID) tokens with traditional dense embeddings can provide a more robust and efficient way to model user behavior at an industrial scale.

TokenMinds extends the PLUM framework to user modeling by utilizing an encoder-decoder architecture. The encoder processes a user's historical sequence of interactions—including long-form and short-form video watches and search queries—to produce dense user embeddings. Simultaneously, the decoder autoregressively generates discrete SID-based user tokens. By using a shared SID vocabulary, the model can unify heterogeneous scenarios (long-form and short-form videos) into a single framework. To handle the computational demands of billions of users, the system employs an asynchronous serving infrastructure that separates the generation of these representations from the real-time ranking process.

TokenMinds demonstrates that discrete SID-based user tokens and dense embeddings provide complementary signals that improve ranking performance. By unifying cross-scenario modeling, the authors report a 50% reduction in training compute and a 31% reduction in serving compute compared to maintaining separate models. Live A/B testing on YouTube surfaces showed statistically significant improvements, with up to 0.11% gains in core user metrics and 0.62% in engagement metrics, confirming the practical viability of tokenized user representations in production environments.

This work provides a blueprint for integrating generative AI techniques into massive-scale industrial recommender systems. By moving beyond fixed-dimensional vectors toward a dual-output paradigm, TokenMinds addresses the limitations of traditional embedding tables while maintaining the compatibility required for existing production ranking pipelines. The ability to unify disparate consumption scenarios into a single model offers a clear path toward more efficient, cross-platform user understanding.