EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

Research Question and Approach

As enterprise agents move from simple text-based assistants to complex tools that interact with files, execute code, and produce business artifacts, evaluating their performance has become increasingly difficult. The authors introduce EnterpriseClawBench, a benchmark designed to capture the realities of workplace agent interactions. Unlike synthetic benchmarks, this project extracts 852 reproducible tasks from actual internal enterprise sessions, including chat logs, tool traces, and file artifacts. The authors developed a pipeline to sanitize, rewrite, and package these sessions into a standardized evaluation format, complete with hard rules for objective success and semantic rubrics for qualitative assessment.

Evaluation Framework

The study emphasizes that evaluating enterprise agents requires a multidimensional approach. Rather than relying on a single score, the authors report results across harness-model combinations, accounting for factors such as artifact delivery quality, cost, runtime, and skill-transfer behavior. The evaluation uses a two-layer scoring system: hard rules verify objective properties (e.g., file existence and format), while LLM-based judges assess semantic dimensions like grounded accuracy and practical utility. The authors also evaluate how well agents can transfer skills learned from one task class to held-out tasks within the same category.

Key Findings

The results reveal that performance is heavily influenced by the interaction between the agent harness and the underlying model. For instance, the same model can perform significantly differently depending on whether it is paired with a harness that supports active environment probing or one that imposes strict approval gates. The authors also observe a non-linear relationship between cost and performance, where diminishing returns occur after reaching a mid-range cost threshold. Furthermore, the study highlights a significant gap in multimodal evaluation: while text-based judging shows strong alignment with human raters, visual judging for artifacts like spreadsheets and presentations remains less calibrated and less reliable.