Little Brains, Big Feats: Exploring Compact Language Models

As Large Language Models (LLMs) continue to dominate AI research, their high computational requirements often make them impractical for on-device deployment or resource-constrained environments. This paper investigates whether Small Language Models (SLMs)—models with significantly fewer parameters—can serve as effective generators within a Retrieval-Augmented Generation (RAG) framework, specifically for Russian-language tasks, without relying on GPU hardware.

The authors constructed a comprehensive Russian-language benchmark by sampling 500 examples from five distinct datasets, including both open-source and proprietary sources. They evaluated 17 different SLMs alongside a state-of-the-art reference model (GPT-5-mini). To ensure a robust evaluation, they implemented an 'LLM-as-a-Judge' framework, selecting three high-performing models (GPT-5-mini, Qwen3-8B, and GLM-4.7) to score the generated responses on correctness, relevance, and faithfulness. All models were tested in a CPU-only environment to simulate real-world, on-device deployment constraints.

The study demonstrates that SLMs are highly capable of performing RAG tasks on standard hardware. Several compact models achieved output quality competitive with larger models while maintaining significantly lower latency. The results highlight that while model size influences performance, specific SLMs provide an optimal trade-off between response quality and computational efficiency. The authors also found that the presence of retrieved context is essential for accuracy, confirming that the models rely on external knowledge rather than just internal parameters.

This research provides a practical roadmap for deploying AI assistants on edge devices, such as personal computers or mobile phones, where privacy and offline functionality are paramount. By proving that SLMs can handle complex RAG tasks without expensive GPU infrastructure, the findings lower the barrier to entry for developers looking to integrate domain-specific, private, and efficient AI solutions into their applications.