Benchmarking Resource-Efficient LLMs for Research Topic Ontology Generation in the Biomedical Field

Knowledge Organization Systems (KOSs), such as ontologies and taxonomies, are essential for structuring scientific knowledge and enabling effective information retrieval. However, the manual curation of these systems is a significant bottleneck, often resulting in outdated or incomplete coverage of rapidly evolving research fields. While Large Language Models (LLMs) offer a scalable alternative for automated ontology generation, their ability to handle the granular, domain-specific semantics required for biomedical classification remains a challenge for smaller, resource-efficient models.

This study evaluates the performance of five open-source LLMs (ranging from 3B to 9B parameters) in identifying semantic relationships between biomedical concepts. The authors introduce MeSH-Rel-4K, a new dataset containing 4,000 semantic relationships extracted from the Medical Subject Headings (MeSH), to benchmark these models across three adaptation strategies: standard prompting, Chain-of-Thought (CoT) prompting, and parameter-efficient fine-tuning.

The researchers formulated the task as a multi-class classification problem, requiring models to categorize pairs of research topics into four types: broader, narrower, same-as, or other. To ensure a robust evaluation, the authors compared standard zero-shot prompting with a two-way CoT approach—which involves reasoning about the relationship in both directions—and fine-tuning using Low-Rank Adaptation (LoRA). The fine-tuning process utilized the Unsloth library to optimize memory usage, allowing for training on consumer-grade hardware.

The experiments demonstrate that while CoT prompting provides a modest baseline improvement over standard prompting (an average F1-score increase of 5.4 percentage points), it is insufficient for expert-level ontology generation. In contrast, targeted fine-tuning leads to substantial performance gains, with the gemma-2-9b-it model achieving an F1-score of 91.6%. Even the smallest model, Llama-3.2-3B-Instruct, showed a 60.5 percentage point increase in F1-score after fine-tuning. These results suggest that fine-tuning effectively overcomes the reasoning bottlenecks inherent in smaller models, providing a viable, automated path for maintaining specialized biomedical ontologies.