Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh

This study investigates whether supervised fine-tuning can improve the ability of small language models (Qwen3.5 at 0.8B, 2B, and 4B parameters) to use statutory context provided at inference time. While retrieval-augmented generation (RAG) is a standard approach for grounding legal AI, small models often fail to correctly apply retrieved provisions. The authors curated a bilingual dataset of 2,165 legal QA records based on six Bangladeshi acts and three schedules. They fine-tuned models using a context-injected approach, where the governing statutory text was included in the training prompt, and then evaluated these models against base versions using the 2022 and 2023 Bangladesh Bar Council examinations.

The researchers found that fine-tuning significantly improves context utilization for the smallest models. For the 0.8B model, fine-tuning increased the 2022 English FAISS-retrieval score from 2 to 34 out of 100. Furthermore, fine-tuning effectively addressed a major issue of language drift, where base models frequently answered Bangla questions in English; the fine-tuned models maintained the requested language with high consistency. However, the benefits of this approach do not scale linearly. While the 0.8B and 2B models showed clear, statistically significant gains, the 4B model exhibited mixed results, with performance improvements in Bangla offset by regressions in several English test conditions.

This research highlights that retrieval quality is not the only bottleneck in legal AI; the model's inherent ability to process and apply retrieved information is equally critical. By demonstrating that context-injected fine-tuning can bridge the gap between retrieval and accurate reasoning, the study provides a practical strategy for deploying smaller, more efficient models in low-resource legal environments. It also underscores the importance of language-specific training to prevent models from defaulting to dominant languages like English when tasked with local legal queries.