A Comprehensive Empirical Evaluation of Vector Database Systems for Approximate Nearest Neighbor Search: Performance, Quality, and Resource Trade-offs

Vector databases have become essential infrastructure for modern artificial intelligence applications such as retrieval-augmented generation (RAG), semantic search, and recommendation systems. These applications rely on dense vector embeddings to represent text, images, and other modalities. However, the ecosystem is highly fragmented, with each database making distinct architectural trade-offs among retrieval quality, query latency, throughput, resource consumption, and operational features.

Previous benchmarking efforts have typically evaluated isolated approximate nearest neighbor libraries rather than full production database systems, or have focused on narrow sets of metrics. This paper addresses this gap by conducting a comprehensive, reproducible empirical evaluation across seven prominent vector database systems—FAISS, Qdrant, Milvus, Weaviate, Chroma, pgvector, and LanceDB—to establish clear guidelines for system selection.

The authors designed a containerized benchmarking framework featuring pluggable database adapters, dataset loaders, metric collectors, and automated experiment runners. The evaluation spans six diverse datasets encompassing over 4 million vectors across computer vision, natural language processing, and synthetic domains, with dimensionalities ranging from 96 to 960.

To ensure fair and reproducible comparisons, each system was deployed on a controlled cloud instance using default, out-of-the-box configurations. The evaluation measured 15 distinct metrics categorized into three main groups:

• Retrieval Quality: Recall at various depths (Recall@K), Precision@10, Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG@100), and Hit Rate.
• Query Performance: Latency percentiles (P50, P90, P95, P99), queries per second (QPS), cold-start latency, and index build time.
• Resource Consumption: Peak RAM utilization, on-disk index size, bytes per vector, and CPU utilization.

The experiments reveal distinct performance profiles and operational trade-offs among the evaluated systems. FAISS achieves the highest single-node throughput (866 QPS on SIFT1M) and superior ranking precision (NDCG), but it functions purely as an in-process library without built-in database operational features like persistence or transaction management.

Among full production databases, Weaviate delivers the highest out-of-the-box retrieval recall (exceeding 99% on SIFT1M) but ranks in the lower half of throughput. Qdrant offers the best latency and throughput balance among full databases, achieving a 4.55 ms median latency and 216 QPS. Milvus excels on high-dimensional data, handling 960-dimensional GIST vectors better than its peers. pgvector integrates vector search seamlessly into PostgreSQL with competitive throughput, though it incurs higher disk storage overhead. Finally, LanceDB trades retrieval quality for significantly faster index construction using a disk-based columnar format and product quantization, while Chroma offers a developer-friendly Python API for rapid prototyping at the cost of the longest index build times.