When Is Complex Chunking Worth It? A Multi-Objective Evaluation of Chunking Methods at Scale

In modern retrieval-augmented generation (RAG) pipelines, long documents must be split into smaller units, or chunks, before being indexed by dense embedding models. While recent research has proposed various advanced chunking strategies—ranging from semantic splitting to large language model (LLM) generated context—most comparisons focus strictly on retrieval accuracy. This leaves crucial operational trade-offs underexplored. The authors investigate whether complex chunking strategies provide retrieval gains that justify their substantial computational costs, and whether these conclusions change as a corpus scales in size.

The study evaluates eight representative chunking strategies (including token-based, sentence-based, late chunking, title-enriched, summary-enriched, contextual, summary-only, and semantic chunking) across multiple axes. The evaluation encompasses three open-source embedding models (Qwen-0.6B, embeddinggemma-300M, and Snowflake-L V2), two scalable corpora (CoRE and KILT with NQ relevance judgments), and corpus sizes ranging from 10 thousand up to 10 million documents. Alongside standard retrieval metrics such as NDCG@10 and Recall@100, the authors measure operational performance including indexing throughput (documents per second), query throughput (queries per second), and peak RAM memory usage during index construction.

The experimental results reveal that computationally expensive chunking methods rarely outperform simpler baselines consistently. While title- and summary-enriched methods can improve single-stage ranking (NDCG@10), simple token and sentence chunking remain highly competitive, especially for first-stage retrieval (Recall@100). Furthermore, methods with similar retrieval performance often differ drastically in system-level costs. For example, LLM-based contextual and summary chunking drastically reduce indexing throughput and demand significantly higher resource overheads. Consequently, the optimal chunking strategy depends heavily on the specific embedding model, dataset, target retrieval metric, and operational deployment constraints.