Amirhossein Abaskohi, Raymond Li, Gaetano Cimino, Peter West, Giuseppe Carenini, Issam H. Laradji
4 min
Abstract
Retrieval-augmented generation (RAG) systems depend critically on how documents are chunked and searched. Fine-grained chunks can improve retrieval precision but expand the search space, increasing latency and cost; larger chunks reduce the number of candidates but make dense similarity less reliable, as the representation for each chunk mixes multiple topics and introduces more semantic noise. This trade-off becomes especially limiting in deep research tasks, where retrieval must be both fast and precise across large, heterogeneous corpora. We introduce MCompassRAG, a metadata-guided retrieval framework that uses topic-level signals as a semantic compass for selecting relevant evidence. Instead of relying only on cosine similarity between queries and noisy chunk embeddings, MCompassRAG enriches chunk representations with topic metadata in the same embedding space and trains a lightweight retriever through LLM-teacher distillation. At inference time, MCompassRAG performs topic-aware retrieval without additional LLM calls, improving both efficiency and evidence quality. Across six complex retrieval benchmarks, MCompassRAG improves information efficiency (IE) by 8.24% on average with over 5 times lower latency than the strongest efficient RAG baselines. Code is available on https://github.com/AmirAbaskohi/MCompassRAG.
Sam: That's a fair concern, and the authors address it directly. The topic information is calculated once, in advance, and stored in what they call a "metadata bank." Because it's pre-computed and cached offline, it doesn't add meaningful overhead to the live search process. When a query comes in, the system just looks up the relevant tags — it doesn't recalculate anything on the fly. So you get the precision benefits without the storage cost you might expect.
Alex: And that's what they mean by "information efficiency" — getting more relevant results without spending more computing power to get them?
Sam: Precisely. The topic-aware approach means each search retrieves more useful chunks per unit of effort. It's a way of bypassing the usual trade-off between chunk size and precision entirely, rather than just accepting it as a fixed constraint.
Alex: How do they decide how many topics to use, though? I'd imagine too few topics and everything is too vague, but too many and it becomes noise.
Sam: That's exactly the tension the paper explores. They found that performance peaks at a moderate level of topic granularity — which is the technical way of saying "how fine-grained your categories are." Think of it like organising a music library. If you only have two genres — "fast" and "slow" — that's not very useful. But if you have ten thousand micro-genres, finding anything becomes chaotic. There's a sweet spot, and the paper identifies that finding it matters for the system to work well.
Alex: It's interesting that the solution here isn't about raw computing power — it's about being smarter with the structure of the information itself.
Sam: That's the central argument of the paper. By treating the topical structure of a document collection as a navigational map, they've shown it's possible to make large-scale retrieval both faster and more accurate at the same time. For tasks that require searching through large bodies of research or documentation, that's a meaningful step forward.
Alex: Thanks for walking us through that, Sam. And thanks to everyone listening to ResearchPod.