Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: $1.69\times$ over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to $2.1\times$ on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about $2\times$ dense throughput while admitting each request with $6.5$--$9.7\times$ less KV and holding $2.2$-$2.6$ less decode-node host memory than full KV transfer.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper about how Large Language Models—those AI systems that power things like coding assistants and chatbots—actually handle memory, and why that turns out to be a surprisingly tricky problem.
Sam: That's right. The system we're discussing is called OasisKV. The puzzle it addresses is this: as we give AI models longer and longer documents to read, they run out of space in their fastest memory. And when that happens, they slow down significantly.
Alex: So the question is how to keep these models running at full speed even when they're processing massive amounts of information?
Sam: Exactly. To understand the problem, think about how these models work. Every time the model reads a word, it builds up a kind of mental map—a record of everything it's seen so far. Researchers call this the "Key-Value cache." Now, for the model to work quickly, that map needs to live in the GPU's fastest memory, which is called High Bandwidth Memory, or HBM. The trouble is, HBM is expensive and limited in size. As documents get longer, that map grows, and eventually the fast memory fills up completely. Researchers call this hitting the "memory wall."
Alex: That's a useful image. It's like trying to write a book report, but your desk is only big enough for five pages at a time. If you constantly have to walk across the room to swap pages in and out of a filing cabinet, you spend more time fetching than actually writing.
Sam: That's exactly it. The HBM is your desk, and the filing cabinet is the slower memory further away. Current systems are stuck choosing between two bad options: they either run out of desk space entirely, or they spend so much time fetching pages that the whole process grinds down.
Alex: So how does OasisKV get around this? Does it just give the system a bigger desk?
Sam: Not exactly—and this is where the approach gets interesting. Instead of expanding the desk, OasisKV acts like a well-prepared librarian. Before starting a new chapter, this librarian glances at a quick summary of what's coming and pulls only the specific pages they'll need. The system doesn't wait until it's desperate for a page; it fetches the right ones in advance. The researchers call this technique "Lookahead Sparse Prefetching."
Alex: So it's predicting what it will need before it needs it. How does it actually know what's coming?
Sam: It uses a technique called "speculative decoding." Here's the idea: the system runs a fast, rough draft of what the next few words are likely to be. This draft isn't the final answer—it's more like a quick sketch. But because it's generated ahead of time, the system can look at that sketch and work out which parts of the filing cabinet it's going to need. Then it moves those specific pieces onto the desk before the real work begins.
Alex: So by the time the model actually needs that information, it's already sitting right there in fast memory?
Sam: Precisely. It turns what was a memory problem into a scheduling problem. The slow part—moving data from filing cabinet to desk—happens in the background, hidden behind the work the model is already doing.
Alex: But what if the sketch is wrong? What if it predicts the wrong pages?
Sam: That's the genuine trade-off here. If the prediction misses, the system has to pause and fetch the correct data the old-fashioned way, which costs time. But the study suggests that for long-context tasks—the kind where this memory problem is most severe—the predictions are accurate enough that the model's output quality stays nearly identical to a system that had all the data available instantly. The cost of occasional wrong guesses is small compared to the gains from getting it right most of the time.
Alex: There's another piece to this, isn't there? Something about how the system is physically set up?
Sam: Right. Modern AI infrastructure often separates the job into two parts running on different machines. One machine handles the heavy initial reading—digesting the full document all at once. A second machine then takes over for the slower, word-by-word generation of the response. The challenge is that the second machine needs access to that Key-Value cache, which lives on the first. In a standard setup, it would have to download the entire cache across the network connection between them—which is slow and creates a bottleneck.
Alex: And OasisKV changes that?
Sam: It does. Because the system already knows which specific pieces of the cache it will need—thanks to that lookahead sketch—it only requests those pieces across the network, not the whole thing. The bottleneck isn't the processing power; it's how fast data can travel between machines. By sending a fraction of the data, they keep that connection from getting overwhelmed. In their tests, this approach achieved more than double the throughput of standard methods on long documents, while keeping accuracy nearly unchanged.
Alex: And this holds up even at very large scales?
Sam: The study tested it on models with hundreds of billions of parameters—these are among the largest AI systems currently in use. The paper suggests the approach scales well, allowing those systems to handle much larger batches of work without requiring additional hardware. The core insight is that you don't need to solve the memory problem by buying more memory. You solve it by being smarter about which memory you move, and when.
Alex: So OasisKV is less about raw power and more about coordination—getting the right information to the right place just before it's needed.
Sam: That's a fair summary. It's a meaningful step toward making long-context AI practical without simply throwing more expensive hardware at the problem. Whether that approach holds up across an even wider range of tasks is something the research community will no doubt continue to examine.
Alex: Thanks for listening to ResearchPod.