Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

Neural relational reasoners often struggle to generalize to problem instances that are more complex than their training data. A primary obstacle to improving these models is the lack of automated, systematic ways to generate challenging test cases, as it is often unclear what structural properties make a reasoning problem difficult. This paper investigates whether Large Language Models (LLMs) can autonomously generate increasingly difficult benchmarks to stress-test and improve these reasoning models.

The researchers propose a framework where an LLM acts as an evolutionary search engine (inspired by FunSearch) or an autonomous agent to discover priority functions. These functions guide the construction of knowledge graphs (KGs) by iteratively adding edges that maximize the failure rate of a target model, such as an Edge Transformer (ET). By treating the model's error rate as a fitness metric, the system evolves samplers that produce progressively harder reasoning tasks. The authors also explore an 'auto-research' agentic loop where an LLM iteratively refines the sampling code to drive down model accuracy while maintaining simplicity.

The study finds that LLM-driven evolutionary search successfully discovers sampling strategies that generate problem instances significantly more challenging than those found in existing benchmarks. These generated instances often expose limitations that are not captured by traditional difficulty metrics like inference depth or off-path edge counts. Furthermore, augmenting the training data with these challenging instances leads to a more robust model (SUPERET) that shows improved generalization. The authors also demonstrate that this machinery is domain-agnostic, successfully applying it to novel reasoning worlds proposed by LLMs.

This work provides a path toward fully autonomous benchmarking for AI reasoning. By replacing manual, heuristic-based dataset creation with automated, model-in-the-loop generation, researchers can more effectively identify the blind spots of neural reasoners and create more systematic evaluation protocols. This approach shifts the burden of finding 'hard' cases from human researchers to automated agents, potentially accelerating the development of models capable of robust, systematic reasoning.