GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

The Problem with LLM-as-Judge

Modern AI agents often operate in complex environments where they must retrieve information from private databases, Slack logs, or time-sensitive records. Current evaluation methods, such as 'LLM-as-judge,' often reward agents for providing a plausible-sounding answer even if that answer was derived from invalid evidence. For example, an agent might correctly identify a churn risk by accessing a report that was created after the question's 'as-of' date, or by accessing a document the user was never permitted to see. Because these agents often produce the 'right' answer, human-like judges frequently overlook the underlying procedural violations.

The GroundEval Approach

GroundEval replaces subjective judgment with a deterministic contract. Instead of asking an LLM if an answer is good, the framework checks the agent's actions against a machine-readable state. This state includes the event log, artifact corpus, access policies, and temporal boundaries. The framework evaluates the agent's trajectory—what it searched, what it fetched, and what it cited—to ensure every step complies with the defined environment. If an agent claims a document does not exist without searching the correct repository, or if it uses information outside its 'visibility cone,' GroundEval flags this as a failure, regardless of whether the final answer appears correct.

Evaluation Tracks

GroundEval categorizes agent failures into three specific tracks:

• Perspective: Tests whether an agent respects temporal and role-based access boundaries, ensuring the agent only uses information available to the actor at the specific time of the query.
• Counterfactual: Evaluates whether an agent correctly identifies cause-and-effect relationships by requiring evidence of specific event links, rather than relying on temporal adjacency or plausible-sounding explanations.
• Silence: Verifies that an agent performs sufficient searches before claiming that information does not exist, preventing 'lazy' negative answers.