Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity

As Vision Language Models (VLMs) become more capable, they are increasingly applied to large-scale document retrieval tasks. However, existing benchmarks often rely on visually diverse documents (e.g., charts, letters, and forms mixed together), which makes retrieval artificially easy because the documents are visually distinct. This paper investigates the performance of retrieval systems in enterprise environments where thousands of documents share nearly identical visual templates—a phenomenon the authors call "embedding collapse."

The authors introduce the Invoice Haystack benchmark, consisting of 1,500 anonymized invoice images and 200 discriminative question-answer pairs. This dataset is specifically designed to be visually homogeneous, with a mean pairwise cosine similarity of 0.73, making it significantly more challenging than previous benchmarks like DocHaystack (0.38) or InfoHaystack (0.31). To solve the retrieval problem in this environment, the authors propose VL-RAG (Vision-Language Retrieval-Augmented Generation). This framework uses a dual-stream approach: a text stream that encodes OCR-extracted content using BGE-Large, and a vision stream that uses SigLIP and OpenCLIP to capture layout and structural features. These scores are fused, and a final VLM-based verification filter is applied to ensure high-precision document identification.

VL-RAG demonstrates superior performance across both the new Invoice Haystack benchmark and existing datasets. On the Invoice Haystack-500 subset, VL-RAG achieves a 60.0% Recall@1, outperforming the previous state-of-the-art by 13.5 percentage points. The authors show that neither vision-only nor text-only approaches are sufficient for homogeneous collections; the dual-stream fusion strategy is essential for distinguishing between documents that share the same template but contain different semantic data.