Train, Retrieve, or Both? A Four-Arm Head-to-Head for Correct Statutory Citation on the Ontario Residential Tenancies Act

This study investigates how to best provide accurate statutory citations for the Ontario Residential Tenancies Act (RTA) and its associated regulations. The authors address a practical challenge: self-represented tenants and landlords often need to identify the specific legal provision governing their situation, rather than receiving fluent but potentially inaccurate legal prose. The researchers conducted a four-arm head-to-head comparison using the Qwen2.5-7B-Instruct model, testing four configurations: a zero-shot base model, a LoRA fine-tuned (SFT) model, a RAG-only model, and an SFT+RAG hybrid.

The study establishes that fine-tuning alone is insufficient for accurate citation, as the SFT-only model frequently mis-recalls specific sections. Retrieval is identified as the critical component for success, effectively reducing hallucinated citations to zero by construction. The SFT+RAG hybrid achieved the highest exact-match score of 0.481. The authors hypothesize that the hybrid approach succeeds because fine-tuning makes the model more robust at selecting the correct provision from the larger, noisier candidate sets provided by the retriever. Notably, the study found that using a lightweight retriever (bge-small) performed as well as or better than more complex pipelines involving larger embedders and cross-encoder rerankers, suggesting that specialized, heavy retrieval models are not necessary for this specific legal corpus.

For developers building legal decision-support tools, this paper provides a clear, empirical baseline for balancing model training and retrieval. It demonstrates that while retrieval is mandatory for grounding, the "best" system does not necessarily require the most computationally expensive components. However, the authors emphasize that their system did not reach the aspirational 0.70 exact-match target, highlighting that retrieval-recall ceilings and difficulty in distinguishing specific subsections remain significant hurdles for automated legal citation.

Key Terms

Statutory Citation

The specific identification of a legal provision, including the Act or regulation name, section number, and subsection.

Exact-Match

A scoring metric where a prediction is considered correct only if the section and subsection match the gold standard, with partial credit given for section-only matches.

Hallucination

The generation of a citation that does not exist within the pinned legal corpus, which the authors mitigate using a deterministic existence-check guardrail.

RAG (Retrieval-Augmented Generation)

A framework where the model is provided with relevant chunks of legal text to ground its responses, rather than relying solely on internal parametric memory.

LoRA (Low-Rank Adaptation)

A parameter-efficient fine-tuning technique used here to adapt the base model to the specific task of mapping questions to statutory citations.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.