Kangqi Chen, Rakesh Nadig, Manos Frouzakis, Nika Mansouri Ghiasi, Yu Liang, Haiyu Mao, Jisung Park, Mohammad Sadrosadati, Onur Mutlu
9 min
Retrieval-Augmented Generation (RAG) is essential for enabling Large Language Models (LLMs) to provide accurate, up-to-date responses by querying external document databases. However, current RAG pipelines are bottlenecked by the retrieval stage, where millions of high-dimensional embeddings must be transferred from storage to the CPU for similarity search. Existing In-Storage Processing (ISP) solutions for this problem often suffer from performance degradation due to irregular data access patterns, lack of efficient document-level retrieval, or the requirement for significant, intrusive hardware modifications.
REIS addresses these issues by treating the retrieval system as an ISP-native framework. It integrates three major innovations: a database layout that links embeddings to document chunks, an ISP-tailored version of the Inverted File (IVF) search algorithm, and an in-storage Approximate Nearest Neighbor Search (ANNS) engine that utilizes existing SSD computational resources.
REIS introduces a specialized database layout that separates embeddings and documents into distinct storage regions, utilizing the Out-Of-Band (OOB) area of NAND flash pages to create low-overhead links between them. This allows the system to identify the relevant document immediately after finding the top-k embeddings without additional host-side I/O.
To perform the search efficiently, REIS adopts the IVF algorithm, which is well-suited for ISP because it allows for streaming-based access patterns. The ANNS engine executes binary quantization to reduce computation complexity and uses distance filtering to discard irrelevant data points early, significantly reducing traffic over the internal flash channels. Crucially, REIS implements these features by leveraging the existing NAND flash peripheral logic, such as pass/fail counters and bitwise XOR operations, avoiding the need for dedicated ASICs or costly hardware modifications.
By moving the retrieval stage entirely into the storage, REIS fundamentally eliminates the I/O bottleneck. Evaluation on both cost-oriented and performance-oriented SSD configurations shows that REIS provides significant speedups and energy savings compared to conventional server-grade systems. Because the system utilizes standard SSD components, it offers a practical, deployable solution for modern LLM inference pipelines that currently struggle with large-scale vector search latency.
Large Language Models (LLMs) face an inherent challenge: their knowledge is confined to the data that they have been trained on.This limitation, combined with the significant cost of retraining renders them incapable of providing up-to-date responses.To overcome these issues, Retrieval-Augmented Generation (RAG) complements the static training-derived knowledge of LLMs with an external knowledge repository.RAG consists of three stages: (i) indexing, which creates a database that facilitates similarity search on text embeddings, (ii) retrieval, which, given a user query, searches and retrieves relevant data from the database and (iii) generation, which uses the user query and the retrieved data to generate a response.The retrieval stage of RAG in particular becomes a significant performance bottleneck in inference pipelines.In this stage, (i) a given user query is mapped to an embedding vector and (ii) an Approximate Nearest Neighbor Search (ANNS) algorithm searches for the most semantically similar embedding vectors in the database to identify relevant items.Due to the large database sizes, ANNS incurs significant data movement overheads between the host and the storage system.To alleviate these overheads, prior works propose In-Storage Processing (ISP) techniques that accelerate ANNS workloads by performing computations inside the storage system.However, existing works that leverage ISP for ANNS (i) employ algorithms that are not tailored to ISP systems, (ii) do not accelerate data retrieval operations for data selected by ANNS, and (iii) introduce significant hardware modifications to the storage system, limiting performance and hindering their adoption.
Alex: Computing inside storage already has a research history. What does REIS change compared with earlier accelerators?
Sam: The paper identifies three gaps. Earlier systems can use search algorithms poorly suited to storage parallelism, neglect fetching the documents after finding their vectors, or add substantial computational hardware. REIS tries to solve those problems together, rather than accelerating the similarity calculation in isolation.
Alex: Let’s take the algorithm first. Why would a search method that works well on a server work poorly inside storage?
Sam: Graph-based methods follow links between vectors. Each visited location helps determine where to go next. That creates irregular accesses and dependencies, which can leave storage’s many parallel resources underused. The authors explicitly find that a strong server algorithm is not automatically the right storage algorithm.
Alex: So they need a search pattern that makes the storage device busy in a more predictable way. What do they choose?
Sam: An inverted-file search, which groups embeddings into clusters. It first finds promising cluster representatives, then searches within those clusters. The embeddings can be read as contiguous streams, making it easier to spread work across the device’s internal resources.
Alex: That explains where to search. What lets the storage hardware actually calculate similarity without a new accelerator?
Sam: Binary quantization represents each embedding component with a single bit. REIS copies the query into buffers inside the flash memory. Existing logic compares those bits with stored embeddings, using an exclusive-or operation that marks where they differ.
Alex: Marking differences is only part of a distance calculation. How do they turn those marks into a usable score?
Sam: They reuse a fail-bit counter, an existing component that counts marked bits. That count gives the distance between the binary vectors. It’s the central hardware example: a component already inside flash becomes part of the search engine.
Alex: Compressing vectors that aggressively could change which documents are selected. How do they protect search quality?
Sam: They retain higher-precision versions of the embeddings too. After the binary search identifies candidates, an embedded processor recalculates their distances at higher precision and reorders them. That refinement is called reranking. The comparisons vary retrieval recall, meaning how many true nearest matches the approximate search recovers.
Alex: Finding a vector still leaves the document somewhere else. How does REIS avoid making that another lookup problem?
Sam: It keeps embeddings and document chunks in separate regions, because embeddings are accessed much more frequently. Each embedding carries its document’s address in spare flash space normally associated with error-correction information. When REIS selects an embedding, the document location is already available.
Alex: So the layout is doing computational work too: parallel vector reads, followed by targeted document retrieval. Does anything else reduce the internal traffic?
Sam: Distance filtering discards candidates that are too far from the query before forwarding them to the storage controller. The evaluation finds this is the most beneficial of the added optimizations. REIS also overlaps reading, distance calculation, and candidate selection, rather than waiting for each stage to finish.
Alex: Flash reads can contain errors. If computation stays inside the flash, doesn’t that bypass the usual correction machinery?
Sam: That is a major design constraint. REIS stores binary embeddings in a more robust single-level-cell region, using enhanced programming to widen the separation between stored values. Documents and higher-precision embeddings use denser storage. The reliability argument relies on that programming technique and the characterization results the paper cites.
Alex: Then “no hardware modifications” does not mean I can run this immediately on any existing drive.
Sam: No. It means the design avoids adding computational hardware. It still needs command-set extensions, controller support, a specialized database layout, and the proposed flash programming behavior. Adoption is a design claim, not a demonstrated software installation on an arbitrary commercial drive.
Alex: With those requirements clear, how broadly did they test the performance claim?
Sam: They evaluate cost-oriented and performance-oriented storage configurations based on commercial products. Their retrieval datasets include Natural Questions, HotpotQA, and Wikipedia collections. The server comparisons use the same cluster-based search approach, with binary compression and reranking, rather than giving only REIS those algorithmic advantages.
Alex: But a server that reloads its data is a demanding baseline. Do they separate that cost from search performance?
Sam: They add an idealized server comparison with no storage-transfer overhead. REIS still outperforms it on average, which they attribute to internal storage parallelism. That strengthens the architectural argument, but it does not establish the same gains for every memory-resident workload.
Alex: What should listeners treat as the main limit on confidence?
Sam: The storage results are modeled and simulated, not measurements from a deployed REIS drive. The authors simulate embedded processors and model flash, memory, and power using existing characterization sources. That is useful architectural evidence, but real integration remains a separate test.
Alex: And even if retrieval improves as predicted, the entire language-model application won’t necessarily speed up by the headline amount.
Sam: The paper’s end-to-end analysis makes that distinction. After REIS removes most retrieval overhead, generation accounts for ninety-two percent of execution time in the reported cases. Larger generation models can also reduce retrieval’s share. Faster retrieval shifts the bottleneck; it does not eliminate generation costs.
Alex: Who should read the full paper, and where would you send them first?
Sam: Researchers building retrieval systems or computational storage should start with Database Layout and the In-Storage ANNS Engine. Then read Methodology and the comparison against the server with no storage-transfer overhead. For deployment questions, read the discussion of maintenance, contiguous storage, and read-intensive workloads. For everyone else, remember this: faster retrieval can come from moving less data, not just calculating faster.
Alex: Keep the data path in view.