Distilling Answer Set Programming Theories from Large Language Models

This paper investigates whether Large Language Models (LLMs) can autonomously author complete, correct Answer Set Programming (ASP) theories from scratch. The authors test if an LLM, provided with only a fixed agent harness and access to an ASP solver, can iteratively refine a theory to solve non-trivial Visual Question Answering (VQA) benchmarks without human-supplied templates or rule patterns.

The researchers developed a dataset-agnostic distillation protocol where an LLM agent is given an empty file and a 1-hour time limit to derive a theory. The agent interacts with a sandbox environment using tools like edit, read, and grep, and can execute the clingo solver to test its theory against training examples. The study evaluates nine models across three VQA benchmarks: CLEVR (compositional reasoning), GQA (real-world images), and CLEVRER (temporal/causal reasoning). The quality of the distilled theories is measured by their accuracy on held-out validation sets.

The study demonstrates that frontier models (Claude Sonnet 4.6, Claude Opus 4.7, and DeepSeek V4 Pro) are highly capable, achieving near-perfect accuracy on CLEVR and exceeding 92% on GQA and CLEVRER. Notably, these models perform consistently regardless of whether they are provided with handwritten reference theories from other domains. In contrast, GPT-5 shows significant performance drops on GQA and CLEVRER, and its accuracy decreases when provided with reference theories, suggesting that context management or model-specific behaviors impact the distillation process. Sub-frontier models generally struggle, often failing to produce syntactically valid ASP code or terminating prematurely.

This work provides a scalable, neurosymbolic pathway for automating the creation of declarative logic programs. By demonstrating that LLMs can act as autonomous theory-builders, the research reduces the high manual cost associated with ASP authoring and suggests that complex symbolic reasoning tasks can be offloaded to LLM agents, provided they are equipped with the right feedback loops and solver integration.