GEM: A Generative Embedding Model Bridging Reasoning and Retrieval

Modern large language models excel at complex reasoning and following detailed instructions, allowing users to express sophisticated information needs. However, conventional retrievers largely rely on surface-level lexical or semantic matching between queries and documents. This creates a growing gap between how users express their intent and how retrieval systems interpret it. While recent methods attempt to bridge this gap by augmenting retrievers with upstream LLM reasoning, they typically rely on multi-stage pipelines and separate models, leaving it unclear whether retrievers genuinely understand the reasoning or merely benefit from increased surface overlap. This paper introduces GEM (Generative Embedding Model) to integrate reasoning and representation learning into a single model.

GEM adopts a generate-then-encode paradigm within a unified architecture. Given a query, GEM first processes a meta-instruction to explicitly reason about the user's intent and relevance criteria, generating a response. It then appends a dedicated embedding token, <|embed|>, at the end of the response. The representation of this token is used for retrieval, with its computation optimized by reusing the key-value cache from the generation phase. To ensure the model aligns its embeddings with its reasoning without suffering from catastrophic forgetting, GEM is jointly trained using causal language modeling and InfoNCE contrastive losses on a curated dataset of filtered reasoning traces and generated positive and hard negative documents.

Evaluated on reasoning-intensive retrieval tasks (such as the BRIGHT benchmark) and instruction-following benchmarks (such as FollowIR and InstructIR), GEM demonstrates robust performance. Using a 4-billion-parameter backbone, GEM outperforms its non-reasoning variant and matches or exceeds baselines using substantially larger models, particularly excelling on reasoning-intensive theorem-based tasks. Furthermore, GEM's generative nature allows users to scale test-time compute through prompting, which further enhances retrieval accuracy.