Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models

This paper introduces B1ade, a resource-efficient Retrieval-Augmented Generation (RAG) architecture consisting of a 335M parameter embedding model (B1ade-embed) and a 1B parameter small language model (B1ade-1B). The authors challenge the conventional assumption that effective RAG requires large-scale pretraining and explicit grounding supervision, proposing instead that strategic model composition and targeted reinforcement learning can achieve competitive performance with significantly lower computational costs.

The authors employ two primary techniques to achieve their results. First, B1ade-embed is constructed using a parameter-free fusion of five pretrained encoders, achieving top-tier performance on MTEB benchmarks without additional training. Second, B1ade-1B is trained using Group Relative Policy Optimization (GRPO) on a curated dataset of 723M tokens. Crucially, the reward function used during training optimizes only for answer similarity (ROUGE-L) and contains no explicit instructions or penalties regarding source citation or grounding.

The study's central contribution is the empirical observation of emergent attribution. Despite the lack of explicit supervision for grounding, B1ade-1B cites retrieved passages in 42.4% of its responses, a 5.5 percentage point increase over the attribution rate present in its training data. This suggests that the model learned to use citations as a strategic tool to maximize accuracy. Furthermore, B1ade-1B demonstrates strong performance on standard QA benchmarks like PopQA (81.82%) and PubMedQA (65.8%), proving that compact models can be highly effective for RAG when trained with optimized, domain-relevant data.

This research provides a blueprint for deploying high-performance RAG systems in resource-constrained environments. By demonstrating that grounding behaviors can emerge naturally through reinforcement learning, the authors suggest that developers can simplify the training process for RAG models, moving away from complex, hand-crafted reward engineering toward more streamlined, accuracy-focused training objectives.