LLM-Based Test Oracles: Source-of-Authority Taxonomy—A Systematic Literature Review

This systematic literature review (SLR) investigates the emerging landscape of LLM-based test oracles. The authors screened 2,436 records to identify 83 relevant studies, analyzing them through a new three-axis taxonomy: the source of the oracle's authority, the form of the oracle, and the adjudication mechanism. By separating these axes, the study addresses a critical gap in the literature, where prior work often conflated the technique used (e.g., LLM-as-a-judge) with the underlying source of truth.

The authors distinguish seven sources of authority, ranging from specification-derived (where an external document defines correctness) to model-parametric (where the model relies solely on its internal training data). A key finding is that specification-independent oracles—those relying on code, prior versions, or the model's own knowledge—comprise over half of the reviewed studies. This independence allows these oracles to function on software that lacks formal documentation, but it also introduces significant trust risks, as these verdicts lack an external, auditable anchor.

Understanding the source of authority is essential for software quality assurance. The review demonstrates that the most frequently reported failure mode is the use of weak oracles that fail to catch bugs, often compounded by model hallucination. Because the source of authority sets the ceiling for how much a verdict can be trusted, practitioners should prioritize oracles with clear, inspectable foundations. The authors argue that future research must move beyond evaluating oracles based on their resemblance to known outputs and instead focus on fault-grounded evaluation, such as mutation analysis, to ensure that generated assertions actually detect intended faults.