Sangjin Choi, Sukmin Cho, Yifan Xiong, Ziyue Yang, Youngjin Kwon, Peng Cheng
5 min
Abstract
In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.
Sam: Correct. It doesn't touch the model weights or alter how the AI thinks. It only changes which hardware worker handles which request. By doing that, the researchers found they could reduce the time it takes to generate each word of a response by a meaningful margin—sometimes over ten percent—without any loss in accuracy.
Alex: But if the router is aggressively sending similar requests to the same workers, doesn't that risk creating a traffic jam? One worker gets swamped while others sit idle?
Sam: That's exactly the right concern. To handle it, they use what they call a "locality-band" approach. Imagine each worker has a home neighborhood of specialists it's good at serving. The router only considers workers within the right neighborhood for a given request. Then, within that neighborhood, it picks whoever has the most capacity. If the best-matched worker is already full, it finds the next best option nearby. So you get the memory benefits of grouping similar requests, without creating bottlenecks.
Alex: It's like assigning students to study groups based on their subject, but making sure no single group gets overcrowded.
Sam: That's a good way to put it. And there's another wrinkle worth mentioning. Sometimes an AI system has already seen a prompt before and stored the result—a bit like a browser cache. The question is whether the routing system can still use the expert signature if it skips the full reading phase. The researchers solve this by saving the signature alongside the cached result. So even when the system skips the initial processing, the router still has the shopping list it needs.
Alex: And do the caching benefits and the routing benefits stack on top of each other?
Sam: They do—the paper suggests the two effects are additive. And as you scale up to larger pools of workers, the gains actually grow. With more workers, the system can split the workload into finer clusters with narrower specialist coverage per worker. That means each worker needs to hold fewer specialists in memory at any given time, which directly translates to lower response latency.
Alex: So the bigger the system, the more this approach pays off?
Sam: That's what the evidence points to. For very large models, the researchers pair this routing method with a technique called "expert parallelism," where specialists are spread across multiple chips. By ensuring each cluster's workload is distributed evenly, they avoid any single specialist becoming a bottleneck. The core insight of the paper is that most research focuses on changing the model itself, but this work shows that how you organize the serving architecture—the traffic control layer—can matter just as much.
Alex: That's a useful reframe. You don't always need a better engine; sometimes you just need better roads.
Sam: Exactly. And the researchers note that the logical next step is making these clusters adaptive—able to reorganize in real-time as traffic patterns shift. For now, the clusters are defined offline, which works well, but live adaptation could make the system even more responsive to unpredictable demand.
Alex: It's a clear demonstration that efficiency gains don't always require rebuilding from scratch. Sometimes a well-designed routing layer is enough. Thanks for walking us through it, Sam.
Sam: Thanks for having me. And thanks to everyone listening to ResearchPod.