ONTOLOGY GENERATION USING LARGE LANGUAGE MODELS | Anna Sofia Lippolis et al. | ResearchPod

Transcript: ONTOLOGY GENERATION USING LARGE LANGUAGE MODELS

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a study that explores how artificial intelligence can help build complex knowledge structures for computer databases.

Sam: So, this paper is basically asking whether a computer can do the heavy lifting of organising information for us?

Alex: Precisely. The core problem is that creating these formal knowledge maps is notoriously tedious and often leads to human error, even among experts.

Sam: And when you say "knowledge maps," you mean a formal way of categorising information — like a digital filing system where every item has a specific definition and a clear relationship to everything else?

Alex: Exactly. The technical term for these is ontologies — structured ways of defining concepts and how they connect to each other. Think of it like building a massive, interconnected dictionary, but designed for a computer to understand rather than a person.

Sam: So if I'm a researcher building a database for, say, a hospital, I have to manually define every single category and every link between them. That sounds like an enormous amount of work.

Alex: It is. And you have to keep everything consistent — one contradictory definition can break the whole system. That's why researchers are now testing whether large language models — the AI systems trained to read and generate human language, like the kind that powers modern chatbots — can draft these structures for us.

Sam: And the paper introduces a new way to instruct those AI models to get better results?

Alex: Yes. They propose a method called Ontogenia. The key idea is that instead of handing the AI one enormous, vague request — "build me a knowledge structure for this domain" — you break the goal down into a series of smaller, focused tasks that build on each other.

Sam: So it's a bit like the difference between telling someone "write me an essay" versus giving them an outline with specific questions to answer one at a time?

Alex: That's a good way to put it. They use what the paper calls "competency questions" — essentially, simple questions that the finished database must be able to answer — to guide the AI step by step. Each question becomes a building block, and the AI assembles the structure piece by piece rather than trying to generate the whole thing at once.

Sam: And how do they check whether the AI actually did a good job? Is there a way to measure the quality of what it produces?

Alex: They use a multi-dimensional evaluation. They check the structure against three specific technical criteria, and then they have human experts review the results to judge whether they hold up in the real world. Both the formal checks and the human judgement matter, because a structure can look correct on paper but still be confusing or misleading to a practitioner.

ONTOLOGY GENERATION USING LARGE LANGUAGE MODELS

Anna Sofia Lippolis, Mohammad Javad Saeedizade, Robin Keskisärkkä, Sara Zuppiroli, Miguel Ceriani, Aldo Gangemi, Eva Blomqvist, Andrea Giovanni Nuzzolese

Mar 7, 2025·3:49·arxiv·Original Paper

Ontology EngineeringLarge Language Models (LLMs)Prompting TechniquesCompetency Questions (CQs)OWL (Web Ontology Language)

Listen

4 min

Briefing

Research Question

Ontology engineering is notoriously difficult, requiring significant time and expertise to translate conceptual requirements into formal structures. This paper investigates whether Large Language Models (LLMs) can bridge this gap by automatically generating Web Ontology Language (OWL) drafts from natural language requirements, specifically user stories and competency questions (CQs).

Approach

The researchers developed and evaluated two novel prompting strategies: Memoryless CQbyCQ, which processes requirements sequentially, and Ontogenia, a more sophisticated approach designed to improve structural coherence. They tested these methods across three different LLMs using a benchmark dataset consisting of 100 competency questions and 29 user stories. To ensure a rigorous assessment, the team employed a multi-dimensional evaluation framework that combines structural metrics with expert qualitative analysis.

Findings

The study demonstrates that LLMs are capable of producing ontologies that meet professional standards. Specifically, the OpenAI o1-preview model, when paired with the Ontogenia prompting technique, produced results that surpassed the modeling capabilities of novice human engineers. However, the authors emphasize that these models are not yet autonomous; they frequently introduce common modeling mistakes and show inconsistent performance across different tasks. Consequently, while LLMs serve as powerful assistants for drafting ontologies, they currently require human intervention to ensure correctness and usability.

Why It Matters

By automating the initial drafting phase of ontology development, this research offers a pathway to significantly reduce the labor-intensive nature of knowledge engineering. As LLMs continue to improve, these techniques could lower the barrier to entry for creating complex, domain-specific ontologies, provided that users remain aware of the inherent risks of model-generated errors.

Key Terms and Definitions

OWL (Web Ontology Language) — A formal language used for defining and instantiating ontologies, allowing for the representation of complex knowledge structures.
Competency Questions (CQs) — Natural language questions that an ontology must be able to answer, used here as the primary requirements for guiding the generation process.
Ontology Engineering — The systematic process of defining the concepts, properties, and relationships that constitute a formal representation of a specific domain.
Memoryless CQbyCQ — A prompting technique where the model processes each competency question independently without retaining context from previous steps.
Ontogenia — A structured prompting framework developed by the authors to improve the logical consistency and structural quality of generated ontologies.

Abstract

The ontology engineering process is complex, time-consuming, and error-prone, even for experienced ontology engineers. In this work, we investigate the potential of Large Language Models (LLMs) to provide effective OWL ontology drafts directly from ontological requirements described using user stories and competency questions. Our main contribution is the presentation and evaluation of two new prompting techniques for automated ontology development: Memoryless CQbyCQ and Ontogenia. We also emphasize the importance of three structural criteria for ontology assessment, alongside expert qualitative evaluation, highlighting the need for a multi-dimensional evaluation in order to capture the quality and usability of the generated ontologies. Our experiments, conducted on a benchmark dataset of ten ontologies with 100 distinct CQs and 29 different user stories, compare the performance of three LLMs using the two prompting techniques. The results demonstrate improvements over the current state-of-the-art in LLM-supported ontology engineering. More specifically, the model OpenAI o1-preview with Ontogenia produces ontologies of sufficient quality to meet the requirements of ontology engineers, significantly outperforming novice ontology engineers in modelling ability. However, we still note some common mistakes and variability of result quality, which is important to take into account when using LLMs for ontology authoring support. We discuss these limitations and propose directions for future research.

ONTOLOGY GENERATION USING LARGE LANGUAGE MODELS: Research Paper Podcast