ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

Modern autonomous agents are increasingly deployed in complex, multi-step workflows, yet current evaluation methods remain largely outcome-oriented. These benchmarks judge only the final state, creating a diagnostic blind spot: they cannot distinguish between an agent that reaches a correct answer through sound, principled reasoning and one that succeeds through lucky guesses or inefficient, fragile paths. This limitation obscures the true reliability of agents and prevents developers from identifying specific reasoning failures, such as poor information synthesis or inadequate self-verification.

ClawTrack addresses this by implementing a dual-assessment benchmark that evaluates both what an agent achieves (Task Score) and how it achieves it (Process Score). The framework consists of 320 tasks across 8 domains, ranging from finance and legal compliance to DevOps and education. Each task is executed in a Dockerized environment with deterministic mock services, producing a structured trajectory of thoughts, actions, and observations.

Evaluation is performed by two independent graders:

• Outcome Grader: Assesses final artifacts, safety compliance, and robustness to tool errors.
• Process Grader: Evaluates each reasoning turn against 12,541 task-specific rubric items across four dimensions: goal alignment, efficiency, information utilization, and result verification.

Evaluating 21 models over 16,000 trials, the authors demonstrate that process scores provide a critical, independent signal of agent quality. Notably, result verification emerges as the most significant systematic bottleneck across all tested models. Furthermore, the authors show that using process-based trajectory filtering—selecting only those paths that are both correct and well-reasoned—leads to consistent post-training improvements, significantly outperforming standard outcome-only filtering methods.