Embedding Inference Attack

Modern Information Retrieval (IR) systems and Retrieval-Augmented Generation (RAG) frameworks rely on embedding models to map text into high-dimensional vector spaces. While these models are often hidden behind APIs, they are vulnerable to various security threats. This paper introduces the Embedding Inference Attack (EIA), a method that allows an adversary to identify the underlying embedding model of a black-box system by analyzing the documents retrieved in response to specific, crafted queries.

The attack operates in a black-box setting where the adversary only observes the set of documents returned by the system, without access to similarity scores or ranking orders. The core idea is that different embedding models produce distinct document retrieval sets for the same query. By comparing the retrieved sets against a known list of candidate models using the Jaccard index, the attacker can iteratively eliminate incorrect candidates until the victim model is identified.

The authors propose several types of adversarial queries to facilitate this identification:

• Random Strings: Sequences of characters that probe parts of the embedding space outside the training distribution of the LLM.
• Non-Queries: Instructions or fragments that force the system to process inputs not structured as standard questions.
• Probing Queries: Direct questions about the system's identity or architecture.

The researchers validated the EIA on both controlled IR pipelines and real-world RAG systems. They found that the attack remains effective even when the target system employs a reranker or uses an LLM that might otherwise reject non-standard inputs. The attack is particularly potent when the number of retrieved documents (k) is high, as this increases the likelihood of observing divergent retrieval results between models.

To defend against EIA, the authors evaluate the use of similarity thresholds, which filter out documents with low relevance scores. While this can effectively reduce the number of discriminative queries available to an attacker, it poses a significant trade-off: aggressive filtering can severely degrade the retrieval quality and overall performance of the IR system for legitimate users.