Can Xiao, Sukmin Cho, Junbong We, Zhixiong Niu, Jianyi Cheng, Yiren Zhao, Youngjin Kwon, Yongqiang Xiong, Rui Ma, Junyi Liu
5 min
Abstract
Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: $1.69\times$ over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to $2.1\times$ on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about $2\times$ dense throughput while admitting each request with $6.5$--$9.7\times$ less KV and holding $2.2$-$2.6$ less decode-node host memory than full KV transfer.
Alex: So by the time the model actually needs that information, it's already sitting right there in fast memory?
Sam: Precisely. It turns what was a memory problem into a scheduling problem. The slow part—moving data from filing cabinet to desk—happens in the background, hidden behind the work the model is already doing.
Alex: But what if the sketch is wrong? What if it predicts the wrong pages?
Sam: That's the genuine trade-off here. If the prediction misses, the system has to pause and fetch the correct data the old-fashioned way, which costs time. But the study suggests that for long-context tasks—the kind where this memory problem is most severe—the predictions are accurate enough that the model's output quality stays nearly identical to a system that had all the data available instantly. The cost of occasional wrong guesses is small compared to the gains from getting it right most of the time.
Alex: There's another piece to this, isn't there? Something about how the system is physically set up?
Sam: Right. Modern AI infrastructure often separates the job into two parts running on different machines. One machine handles the heavy initial reading—digesting the full document all at once. A second machine then takes over for the slower, word-by-word generation of the response. The challenge is that the second machine needs access to that Key-Value cache, which lives on the first. In a standard setup, it would have to download the entire cache across the network connection between them—which is slow and creates a bottleneck.
Alex: And OasisKV changes that?
Sam: It does. Because the system already knows which specific pieces of the cache it will need—thanks to that lookahead sketch—it only requests those pieces across the network, not the whole thing. The bottleneck isn't the processing power; it's how fast data can travel between machines. By sending a fraction of the data, they keep that connection from getting overwhelmed. In their tests, this approach achieved more than double the throughput of standard methods on long documents, while keeping accuracy nearly unchanged.
Alex: And this holds up even at very large scales?
Sam: The study tested it on models with hundreds of billions of parameters—these are among the largest AI systems currently in use. The paper suggests the approach scales well, allowing those systems to handle much larger batches of work without requiring additional hardware. The core insight is that you don't need to solve the memory problem by buying more memory. You solve it by being smarter about which memory you move, and when.
Alex: So OasisKV is less about raw power and more about coordination—getting the right information to the right place just before it's needed.
Sam: That's a fair summary. It's a meaningful step toward making long-context AI practical without simply throwing more expensive hardware at the problem. Whether that approach holds up across an even wider range of tasks is something the research community will no doubt continue to examine.
Alex: Thanks for listening to ResearchPod.