Kalypso: Relational LLM Serving

Modern data management systems increasingly use Large Language Models (LLMs) as semantic operators—such as filtering, joining, or transforming data—to process unstructured information. Existing systems typically treat these operators as independent requests, failing to account for the structure of the overall query plan. This leads to redundant computations, specifically the re-computation of Key-Value (KV) cache prefixes when intermediate results are materialized between operators.

Kalypso introduces the concept of relational LLM serving, which exposes the semantic query plan to the serving layer. By understanding the dependencies between operators, Kalypso can pipeline execution. When a tuple flows from one operator to the next, the system retains the KV-cache state of the shared prompt prefix, allowing downstream operators to skip the expensive prefill phase. This is a significant departure from traditional operator-at-a-time execution, which often forces the system to evict cached prefixes under memory pressure.

To maximize performance, Kalypso employs an adaptive, memory-aware scheduling algorithm. The scheduler must balance two competing goals: maintaining high GPU utilization through parallel execution and preventing the eviction of valuable KV-cache prefixes. It continuously monitors GPU memory occupancy and dynamically adjusts the number of concurrent operator instances. If memory pressure rises, the scheduler throttles upstream operators to ensure that downstream operators have the necessary cache space to complete their tasks without recomputing prefixes.

As LLMs become standard tools for data processing, the efficiency of executing complex semantic queries becomes a primary bottleneck. Kalypso demonstrates that query-aware serving is not just an optimization but a necessary architectural shift. By effectively managing the KV cache as a shared resource across a query pipeline, Kalypso achieves substantial speedups without requiring changes to the underlying LLM models or sacrificing output accuracy.