OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Root cause analysis (RCA) is a critical test for LLM agents, requiring them to reason about complex system failures using heterogeneous telemetry like traces, metrics, and logs. Current evaluation methods focus almost exclusively on whether an agent correctly identifies the root-cause component. This paper investigates whether these agents are actually reasoning through the causal propagation of a fault or simply performing pattern matching, by introducing a process-level evaluation framework.

The authors introduce PAVE (Path Annotation via Verified Effects), a labeling protocol that uses fault-injection records to reconstruct the actual causal propagation path of a failure. Unlike standard approaches that rely on backward inference, PAVE uses forward verification: it checks which downstream anomalies are statistically and temporally consistent with a known intervention. By applying PAVE to three distinct microservice systems, the authors created OpenRCA 2.0, a benchmark containing 500 instances with verified step-wise causal annotations.

The study reveals a significant gap between outcome-based success and process-level reasoning. While LLMs identify at least one correct root-cause service in 76.0% of cases, they only ground that service in a verified causal path in 61.5% of cases. This discrepancy—the ungrounded diagnosis—is hidden by traditional outcome-only metrics. Furthermore, the authors find that agents are much better at identifying the correct services (Node F1) than they are at correctly mapping the directed dependencies between them (Edge F1), suggesting that the agents' internal reasoning chains are often disconnected from the actual causal structure of the system.

This research demonstrates that current RCA benchmarks are insufficient for measuring the true reasoning capabilities of LLM agents. By showing that agents can arrive at correct answers for the wrong reasons, the authors argue that process-level ground truth is essential for developing trustworthy diagnostic tools. This work provides a new standard for evaluating whether an agent's diagnosis is supported by a logically sound, verified causal explanation.