Building a scalable and real-time recommendation system is vital for many businesses driven by time-sensitive customer feedback, such as short-videos ranking or online ads. Despite the ubiquitous adoption of production-scale deep learning frameworks like TensorFlow or PyTorch, these general-purpose frameworks fall short of business demands in recommendation scenarios for various reasons: on one hand, tweaking systems based on static parameters and dense computations for recommendation with dynamic and sparse features is detrimental to model quality; on the other hand, such frameworks are designed with batch-training stage and serving stage completely separated, preventing the model from interacting with customer feedback in real-time. These issues led us to reexamine traditional approaches and explore radically different design choices. In this paper, we present Monolith, a system tailored for online training. Our design has been driven by observations of our application workloads and production environment that reflects a marked departure from other recommendations systems. Our contributions are manifold: first, we crafted a collisionless embedding table with optimizations such as expirable embeddings and frequency filtering to reduce its memory footprint; second, we provide an production-ready online training architecture with high fault-tolerance; finally, we proved that system reliability could be traded-off for real-time learning. Monolith has successfully landed in the BytePlus Recommend product.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at recommendation systems—the kind that suggest videos or ads based on what you've watched before.
Sam: These systems power apps like short-video platforms, where trends go viral hourly. But standard deep learning tools, designed for things like image recognition, struggle here because user data changes so fast, and most of it is sparse.
Alex: Sparse? You mean like mostly empty?
Sam: Right. Imagine user data as a huge spreadsheet. Most cells are blank or filled with rare categories, like a specific video ID that only a few people watch. To use this in AI models, teams turn those categories into lists of numbers—a compact summary that captures what makes that video unique. They store these summaries in a big lookup table, kind of like a phone book where IDs point to their number lists.
Alex: Okay, that makes sense—like a dictionary for the AI. But with billions of users and videos, that table gets massive.
Sam: Exactly. It won't fit in one computer's memory, and it keeps growing as new users join. A common fix is hashing: you take an ID, run it through a math recipe to pick a spot in the table quickly, like folding a map to find an address fast. But if two IDs fold to the same spot—a collision—they share the same summary, mixing up popular videos with rare ones and hurting suggestions.
Alex: So collisions quietly make recommendations worse as the system scales? And user tastes shift minute to minute anyway.
Sam: Yes, that's concept drift—when what people like changes rapidly, like switching from cat videos to dance challenges overnight. Old models trained on yesterday's data push stale content. This work introduces Monolith, a system with a collisionless lookup table that avoids those shared spots entirely, using two math recipes per ID and smart eviction if needed—like hotel rooms where guests have two keys and politely relocate if occupied. It pairs this with real-time updates from fresh user feedback.
Alex: Huh. So it keeps unique profiles for everything without wasting space, adapting on the fly. Right, but how does the system actually handle all those updates without crashing under the load? With billions of entries changing by the minute.
Sam: It uses a setup where different computers have specific jobs. Some machines do the heavy math calculations based on the latest user actions—like workers crunching numbers on fresh data streams. Others act as storage hubs, holding the model's knowledge and updating it safely when workers send changes. This is called a worker-parameter server architecture.
Alex: Okay, so the workers handle the thinking, and the servers keep the memory straight. But what's stored where, exactly?
Sam: The model splits its information into two main types. One type is a network of connected numbers that weigh different factors together, like a brain adjusting how much to care about past likes versus recent views. These are dense parameters, updated less often since they change smoothly. The other type maps unique labels—like a specific user or video—to short lists of numbers summarizing their traits; these are sparse parameters, mostly empty space with bursts of activity for popular items.
Alex: Sparse makes sense for all those rare videos no one's seen yet. So how do they keep memory from exploding with every new one?
Sam: They watch real patterns: most labels show up just a few times, like unpopular words in a dictionary that barely help predictions. Keeping them wastes space without boosting accuracy, since there's not enough data to learn from them. They also drop old labels from inactive users or outdated videos that no longer matter. To do this, they set rules like minimum appearance counts before adding a label, plus a quick check system that sometimes skips low-value ones probabilistically—these are filtering heuristics.
Alex: Huh, so it's like pruning an overgrown garden—cut the weeds and dead branches to focus on what's useful now. How do they turn raw user clicks into usable training data so fast?
Sam: User actions—like clicks or likes—get logged into one stream of messages, while details about videos or users flow in another stream. A processing job matches them up in real time, pairing a click with the right video info to make a complete training example. They use tools like message queues for the logs and a stream processor for the matching; the paper calls this a streaming engine with an online joiner. They keep recent matches in fast memory cache, like a quick notepad. For older ones still waiting, they spill to disk storage, checking there if needed, so lookups stay speedy.
Alex: Matching streams sounds tricky if clicks don't arrive in perfect order. Like if someone clicks a video days later—how do they connect it without losing data or bloating memory?
Sam: To balance uneven data—way more skips than likes—they sample negatives selectively during joining, then adjust predictions later to avoid bias. Workers update a training storage hub with changes from touched items—only the sparse summaries that got hit recently, tracked in a simple list of keys. Every minute, they push just that small changed batch to the serving hub, light on network and memory. Dense parts update less often, like daily, since they shift slowly; the paper notes any staleness there is tolerable with their optimizers, with no clear quality drop.
Alex: Huh—so sparse gets priority because that's where the action is, keeping the loop tight. Does the paper back this up with actual tests on how much better it performs?
Sam: They ran experiments on two datasets to check the collisionless table's impact. One is a public set of movie ratings turned into yes/no likes for realism. Models using collided lookups scored lower on a key measure of suggestion quality—how well it ranks good matches over bad ones, like sorting hits from misses in a game. Researchers call this AUC. The collision-free version consistently reached higher scores, even after many training rounds, and stayed strong as tastes shifted over days. On their live system, they compared hashing to the collisionless approach. Again, the clean version lifted online performance steadily, despite daily fluctuations from shifting user interests. No signs of overfitting.
Alex: Higher scores across time and training—that suggests it handles real-world changes without falling apart. And for the real-time updates—did they test different speeds?
Sam: To probe update frequency, they simulated on ad-click data: batch-train first, then online shards mimicking hourly or half-hour syncs. Frequent pushes kept quality up as new data rolled in. A live A/B split on real ad traffic pitted online against batch-only, confirming the edge in adapting fast. Workers push only changed sparse parts minute-by-minute to serving storage, keeping loads light. For crashes, they snapshot full training storage daily—a full copy to disk. Tests show one day's loss barely dents quality, thanks to smooth-updating dense parts via momentum tools that carry progress forward. The paper notes minimal user impact.
Alex: Tolerable trade-offs for scale. Pulling it all together, it seems like this setup handles the quirks of recommendation data—those sparse traits that pop up rarely and shift fast.
Sam: Yes. The system manages sparse categorical features—unique labels for users or items that appear infrequently—by filtering out the rarest and oldest ones with simple rules like minimum counts. This keeps memory in check without losing much quality, while the real-time loop from user actions feeds recent feedback directly back in, easing concept drift as tastes evolve hourly. The paper shows online updates lift suggestion accuracy over batch methods, with more frequent syncs pulling ahead steadily. The evidence from datasets and live tests points to reliable gains in adapting to dynamic user patterns. It's a meaningful foundation for systems facing nonstop change.
Alex: A solid, scalable way forward without overcomplicating things. Thanks for joining ResearchPod.