ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents

Industrial tool-use agents often fail in ways that are obscured by simple outcome-based evaluation. The authors ask how to build a diagnostic framework that can localize capability deficiencies within a single request execution while simultaneously ensuring that service remains consistent and responsive across long-term user interactions.

ATLAS (Ability Taxonomy and Logical Assessment System) introduces a dual-horizon evaluation structure. Within-turn evaluation uses a matrix of trajectory-wise diagnostic signals to assess execution dimensions (Thinking & Reflection, Tool & Skill Execution, Response Generation) against capability concerns (Relevance, Factuality, Timeliness, Reliability, Intent & Planning). Across-turn evaluation uses user-wise signals to monitor whether the agent maintains context, intent, and constraints over multiple turns. These signals are instantiated as executable, calibrated LLM judges or rule-based checks. To manage costs, the authors distill these judges into efficient diagnostic models and use the resulting signals as multi-dimensional feedback for policy optimization via Group Relative Policy Optimization (GRPO).

ATLAS significantly improves agent performance in production settings. In offline replay experiments, the ATLAS-optimized policy outperformed the baseline across all execution dimensions and user-wise metrics. Online A/B testing on Meituan Xiaotuan showed that this optimization led to a 6.9% increase in session follow-up rates and a 7.32% increase in paid gross transaction volume (GTV), alongside measurable reductions in hallucination rates and improvements in response-supply relevance.

This work shifts agent evaluation from aggregate performance reporting to actionable diagnostic evidence. By linking evaluation signals directly to policy optimization, the framework allows development teams to treat capability deficiencies as explicit improvement targets, creating a closed-loop system that is both interpretable and effective for complex, real-world industrial applications.