SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation

As autonomous agents perform increasingly complex, multi-step tasks, their execution traces often exceed the context windows of even the most advanced Large Language Models (LLMs). This creates a significant challenge for fault attribution—the process of identifying which specific step caused a system failure. The authors investigate how to accurately diagnose these faults without relying on passive, linear ingestion of entire trajectories, which suffers from attention dilution and context limits.

The authors introduce SAFARI (Scaling long-horizon Agentic Fault AttRibution via active Investigation). Instead of loading a full trace into context, SAFARI employs an 'Investigator Agent' that interacts with the trajectory through a specialized toolbox. This agent uses read and search tools to selectively query segments of the trace, mimicking a human developer debugging logs.

To maintain coherence across long investigations, SAFARI utilizes a persistent Short-Term Memory (STM) that stores task goals, hypotheses, and investigative gaps. A critical component of the framework is the verification stage, where the agent decomposes its findings into atomic claims. These claims are then validated by a separate Reasoning Evaluator LLM, which checks if the cited evidence supports the conclusion, ensuring the diagnostic process remains grounded.

SAFARI significantly outperforms state-of-the-art baselines in resource-constrained environments. On the Who&When benchmark, it exceeds existing performance by 20% within a 1M token budget. More impressively, on the TRAIL GAIA subset, SAFARI maintains a 0.58 precision even when the target fault is located 5x beyond the model's native context window—a scenario where traditional, context-dependent evaluators fail entirely. The results suggest that active, hypothesis-driven exploration is a more robust strategy for long-horizon diagnostics than passive context ingestion.