AutoTraceGT: Using Grounded Theory for Agent Behavior Analysis at Scale

Understanding why autonomous agents fail in complex, multi-step tasks is a significant challenge. Traditional quantitative metrics like success rates are process-blind, while manual qualitative analysis is too labor-intensive to scale. AutoTraceGT addresses this by operationalizing Grounded Theory—a rigorous, inductive qualitative methodology from the social sciences—into an automated, multi-agent pipeline. The system uses four specialized agents to perform open, axial, and theoretical coding, iteratively refining a behavioral codebook until it reaches theoretical saturation, where new data no longer yields new structural insights.

The AutoTraceGT pipeline functions through a continuous loop of data sampling and constant comparison. The OpenCode agent labels individual trajectory segments with conceptual codes, while the AxialCode agent groups these into behavioral categories and identifies relationships. A central Manager agent oversees this process, maintaining an auditable revision log and applying a merge-first policy to consolidate patterns. This iterative process continues until the number of new categories added per round falls below a predefined threshold, signaling that the theory is saturated and stable.

Across six diverse trajectory corpora, AutoTraceGT demonstrated high process reliability and artifact quality. The induced codebooks were found to be stable across independent runs and consistent across different backbone LLMs, indicating that the system recovers dataset-specific behavioral structure rather than model-specific noise. When compared against expert-constructed human taxonomies, AutoTraceGT recovered the vast majority of known failure modes and surfaced additional, systematic patterns that human analysts had previously missed. Furthermore, the codebooks can be repurposed as deductive feature spaces, outperforming few-shot LLM baselines in downstream failure prediction tasks.