SUB-BILLION, SUPER-FRONTIER: SMALL LANGUAGE MODELS RIVAL ZERO-SHOT FRONTIER LLMS ON GENERAL AND LITERARY RELATION EXTRACTION

This study investigates whether compact, sub-billion parameter language models (SLMs) can bridge the performance gap with massive, proprietary frontier LLMs in the domain of relation extraction (RE). While frontier models excel at zero-shot tasks, their high computational costs and reliance on proprietary APIs limit their use in resource-constrained or privacy-sensitive environments. The authors evaluate five SLMs (360M to 3B parameters) across 30 different configurations, testing various domain-composition regimes and prompt-conditioned tuning styles. They compare these against zero-shot frontier LLMs and a discriminative RoBERTa baseline across nine benchmarks spanning general-domain and literary texts.

The results demonstrate that targeted task adaptation allows compact models to rival or exceed the performance of much larger systems. Specifically, the Qwen2.5-0.5B model, when fine-tuned on pooled general-domain data, achieved a micro-F1 score of 0.83, outperforming GPT-5.4 (0.69) and Claude Sonnet 4.6 (0.66) in a zero-shot setting. In the literary domain, the gap was even more pronounced, with tuned SLMs outperforming GPT-5.4 by significant margins on benchmarks like the Biographical corpus. The authors clarify that this success is not due to SLMs being inherently more powerful, but rather the effectiveness of task-specific fine-tuning compared to zero-shot prompting.

This research provides a practical roadmap for deploying efficient, private, and high-performance RE systems on consumer-grade hardware. By showing that task-specific fine-tuning is more effective than relying on the zero-shot capabilities of massive models, the study offers a path for organizations to build specialized NLP pipelines without the infrastructure burden or privacy risks associated with large-scale proprietary APIs. It highlights that data-centric optimization and parameter-efficient fine-tuning (PEFT) can effectively democratize access to advanced information extraction.