Evaluating RAG for French immigration law: a benchmark and baseline study

International recruitment in France involves navigating a complex, multi-layered legal framework consisting of immigration laws, labor regulations, and bilateral agreements. Existing legal AI benchmarks primarily focus on common law systems and do not address the specific administrative guidance required for French immigration. This paper introduces a new benchmark to evaluate how well AI systems can recommend residence permits, identify required documents, and provide accurate legal citations for foreign worker recruitment.

The authors constructed a benchmark consisting of 52 annotated synthetic applicant profiles, designed to cover diverse legal regimes and permit categories. They evaluated two scales of the Qwen3.5 model (9B and 27B parameters) using two configurations: a parametric-only baseline (LLM-alone) and a Retrieval-Augmented Generation (RAG) approach that provides the model with relevant legal context via dense vector search. Performance was measured across three dimensions: permit-type recommendation, document requirements, and legal citation coverage.

The study demonstrates that retrieval grounding is essential for reliable administrative guidance in this domain. RAG significantly boosted permit-type accuracy for both model scales, with the 27B model reaching 75% accuracy compared to 50% for the baseline. While RAG also improved document prediction and legal citation recovery, these gains were more pronounced in the larger 27B model. The authors observed that retrieval helps models avoid common errors related to identifying the correct legal regime, though it does not fully eliminate hallucinations or procedural omissions.

This work highlights the persistent difficulty of applying LLMs to highly regulated administrative tasks where accuracy and legal grounding are paramount. By providing a public benchmark for the French immigration context, the authors offer a foundation for future research into hybrid retrieval strategies and more robust legal reasoning systems. The findings suggest that while RAG is a necessary component for legal AI, current models still struggle with the nuanced, multi-source reasoning required to provide exhaustive administrative guidance.