ResearchPod Summary
Modern large language model (LLM) serving often uses disaggregated architectures, where prefill (prompt processing) and decode (token generation) phases run on separate GPU pools. This separation prevents resource contention but introduces a new bottleneck: under bursty workloads, prefill nodes become saturated while decode nodes remain underutilized. In these scenarios, the time spent waiting in prefill queues and transferring the KV cache between nodes dominates the Time-to-First-Token (TTFT), often accounting for over 75% of the total latency.
Kairos introduces a proactive scheduling layer that allows decode nodes to absorb prefill work. Instead of forcing all prefills through the saturated prefill pool, Kairos uses a lightweight, state-aware algorithm to determine if a request can be deflected to a decode node. This deflection executes the prefill phase as chunked-prefill steps interleaved with ongoing decode batches. By building the KV cache directly on the decode node, Kairos eliminates the costly inter-node KV cache transfer entirely.
To ensure that deflecting prefill work does not degrade the performance of existing decode requests, Kairos employs:
By effectively utilizing the idle compute capacity on decode nodes, Kairos significantly improves tail latency and SLO attainment in production-style environments. It allows clusters to handle bursty traffic more gracefully without requiring additional hardware, providing a more efficient path for scaling LLM inference services.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.