When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

Competency Questions (CQs) are essential requirements used to evaluate whether an ontology correctly models its intended domain. The process of CQ-verification—checking if an ontology can answer these natural language questions—is often time-consuming and prone to errors. As ontology engineering becomes increasingly automated through the use of Large Language Models (LLMs), understanding the factors that make CQs difficult to interpret or verify is critical for improving the reliability of the entire development lifecycle.

This study builds upon the OE-Assist framework, a tool designed to support semi-automatic CQ-verification. The researchers analyzed data from 19 ontology engineers who performed 20 verification tasks. The analysis combined quantitative metrics—such as decision duration, perceived task difficulty, and readability scores (Flesch–Kincaid and Gunning Fog)—with qualitative feedback from user session notes. The goal was to identify specific pitfalls in CQ formulation that lead to inconsistent verification outcomes.

The empirical analysis revealed that while decision duration is a strong predictor of perceived task difficulty, other factors like ontology size or automated readability metrics did not show a statistically significant correlation with overall correctness. However, qualitative analysis highlighted three major pain points: syntactically incorrect LLM-generated SPARQL queries, high linguistic complexity, and semantic ambiguity. Specifically, the study found that terms like "resource" are prone to cross-linguistic interference, where different interpretations across languages (e.g., financial means vs. source) lead to confusion. The authors conclude that readability metrics are useful for flagging complex questions and that explicit conceptual clarification is required to reduce ambiguity.