SMETRIC: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

The Challenge of Agentic Serving

Modern LLM applications, such as coding agents, operate differently than traditional human-centric chat. Agents generate long sequences of requests within a single session, and they act only on complete responses rather than streaming tokens. This shift makes the cluster's total tokens per second (TPS) the primary performance metric. Furthermore, agentic workloads exhibit high KV cache (KV$) reuse—often exceeding 80%—because follow-up requests in a session frequently build upon the context of previous turns. Existing schedulers often prioritize local cache hits, which leads to imbalanced clusters where a few instances are overloaded while others remain idle, ultimately capping the cluster's throughput.

The SMETRIC Approach

SMETRIC introduces a balanced, session-centric scheduling policy designed to reconcile the tension between load balancing and cache efficiency. The core insight is that the cluster can achieve both goals by treating the first request of a session differently from follow-up requests.

• First-turn requests: These are routed purely to balance the load across the cluster. Because these requests often share common system prompts, they can fetch necessary KV data from a global-tier store if it is not available locally.
• Follow-up requests: These are routed using traditional cache-aware logic. Since follow-up requests typically reuse the KV state generated by the previous turn in the same session, they are sent to the instance that served the earlier part of the session, preserving local cache hits and minimizing demand on the global tier.

This approach is stateless; the router does not need to maintain a persistent record of user sessions. Instead, it uses request metadata to identify the session turn, keeping the scheduler simple and scalable.

By effectively balancing the load without sacrificing the benefits of local KV caching, SMETRIC addresses a critical bottleneck in large-scale agentic serving. Evaluations on production traces show that SMETRIC significantly improves TPS and reduces latency compared to state-of-the-art schedulers. This allows model providers to serve more agentic traffic with the same infrastructure, reducing costs and improving the responsiveness of complex agentic workflows.