Zhekai Chen, Chengqi Duan, Kaiyue Sun, Bohao Li, Yuqing Wang, Manyuan Zhang, Xihui Liu
5 min
As large language models evolve into proactive agents capable of operating tools and assisting users in real-world environments, existing benchmarks have become insufficient. They often rely on static, sandboxed environments and single-turn evaluations, which fail to capture the iterative, multi-turn nature of real-world human-agent interaction. This paper asks: How can we systematically measure the true capabilities of proactive agents in dynamic settings, and what factors—base model versus agent framework—most influence their success?
The authors introduce UniClawBench, a benchmark featuring 400 bilingual tasks categorized by five foundational capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Unlike previous benchmarks, UniClawBench operates within live Docker containers, using real software and browsers. To solve the challenge of evaluating dynamic tasks without leaking ground truth, the authors implement a three-role closed-loop evaluation strategy: an executor agent performs the task, a hidden supervisor agent evaluates progress using fine-grained, step-by-step rubrics, and a user simulator provides natural language feedback based only on visible evidence.
The researchers evaluated state-of-the-art models across three different agent frameworks (OpenClaw, EDICT, and Nanobot). The results demonstrate that the agent framework design exerts a stronger influence on task performance than the base model itself. Furthermore, the study identifies long-context reasoning and multimodal understanding as the primary bottlenecks for current proactive agents, highlighting that even powerful models struggle when required to synthesize information across diverse, real-world sources.
By disentangling base model capabilities from framework-level design choices, UniClawBench provides a diagnostic tool for developers to pinpoint exactly why an agent fails. This capability-driven approach moves the field away from opaque, scenario-based benchmarks toward a more modular understanding of agent architecture, which is essential for building reliable, long-term personal assistants.
The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.
Alex: So the big finding is that the framework matters more than the model itself?
Sam: That's what the study suggests. Even a very capable model can fail if its framework — the way it plans and sequences its actions — is inefficient. How you organize the work turns out to be just as important as the raw intelligence of the model doing it.
Alex: If we keep the same model but change the system around it, do we actually see different results?
Sam: Consistently. They found that a centralized system — one that keeps all history, context, and feedback in a single place — outperformed fragmented systems where different parts of the task were handled by separate components. The reason comes down to memory. When you split the work across multiple agents, information gets dropped during handoffs. Each piece of the system only knows its own slice of the story.
Alex: So it's like a game of telephone. By the time the instruction reaches the last player, something important has been lost.
Sam: Exactly. The centralized approach keeps the full context unified, so the model can make decisions with the complete picture in front of it. That's why the architecture of the workflow — not just the intelligence of the model — is the primary bottleneck for real-world reliability.
Alex: So we've established that the framework is the real engine here. But is this system itself without flaws?
Sam: Far from it. The researchers are candid about the limitations. The benchmark is built on around 400 manually curated tasks. That's a reasonable starting point, but it's a relatively small sample that may not capture every real-world scenario an agent might face.
Alex: And I imagine using other AI models as the supervisors introduces its own problems?
Sam: It does. If the supervisor model has its own blind spots or biases, it might unfairly penalize or reward the agent being tested — which could skew the entire evaluation. It's a bit like having a teacher who grades based on personal preference rather than the actual quality of the work.
Alex: And I'd guess the live environment creates its own instability. If a website changes its layout overnight, the agent might fail a task through no fault of its own.
Sam: That's a genuine challenge. It becomes very difficult to distinguish between a real agent error and an external glitch. The authors are also open about the risk that developers might try to game the benchmark — over-optimizing for these specific tasks rather than building genuinely capable agents.
Alex: So the goal isn't to create a perfect test. It's to move toward more realistic, transparent evaluations.
Sam: That's the core contribution. By identifying where agents break down and why, the paper provides a clearer roadmap for future development. One direction the authors point toward is agents capable of something like self-diagnostics — automatically identifying their own capability gaps and triggering targeted training to address them.
Alex: That would be a meaningful step forward — moving from agents that just fail silently to ones that can explain what went wrong. Thanks for walking us through this, Sam. And thanks to everyone listening to ResearchPod.