Jun Zhao, Leiming Fu, Yanbo Wen, Yiding Wang, Xuantong Liu, Yang Shu, Yuyang Lu, Xuanran Xing, Jingqi Tong, Hao Xu, Qi Zhang, Xuanjing Huang
8 min
Abstract
Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Five frontier LLMs operate along continuous trajectories under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. We evaluate them through both realized outcomes and mechanism-specific diagnostics derived from complete decision traces. Across 30 days of live evaluation, we find a pronounced outcome-capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use. Trace-level diagnostics further expose distinct bottlenecks across capabilities, demonstrating that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability. LiveMACEBench makes this distinction measurable, turning live markets from a performance leaderboard into a diagnostic environment for agent capability
Alex: That distinction organizes the whole benchmark. Take rule following. The agent receives hard constraints and lower-priority preferences, with an explicit hierarchy. The evaluation checks both whether its actions comply and whether its explanation demonstrates correct, state-grounded rule reasoning.
Sam: Give me the comparison where looking only at the result would lead me astray.
Alex: Under rule guidance, GPT and DeepSeek finished with almost identical returns. But GPT passed the hard-rule checks about seventy percent of the time, versus roughly ninety percent for DeepSeek. Similar profits concealed quite different compliance.
Sam: That makes the practical distinction clear. If I need a constrained agent, matching another system’s profit doesn’t establish that it respects the constraints.
Alex: And compliance alone doesn’t establish good explanations. DeepSeek’s executed decisions often passed, but its written reasoning was less auditable. Gemini performed strongly on both execution and explanations, despite earning less than the return leaders.
Sam: Were the failures mostly agents ignoring instructions, or could the traces locate something more specific?
Alex: A recurring problem was the limit on exposure to a single asset. Some agents incorrectly combined existing holdings, proposed orders, and price changes when estimating their position after a trade. Recognizing the rule was not enough; they also needed an accurate account of their state.
Sam: That’s a more useful diagnosis than saying a model is weak at instruction following. Does the same separation show up with tools?
Alex: Yes. The evaluated tool-use agents generally gathered enough information, but invocation efficiency was their weakest judged dimension. They could reach adequate coverage through redundant interactions. The authors also separated finding an appropriate tool from successfully calling it.
Sam: Because a system could discover the right source and still fail at execution?
Alex: That happened. Qwen was relatively weak at tool discovery but reliable once a tool was available. Grok showed the opposite pattern. GPT was strong at both stages, yet Grok earned the highest return in that configuration.
Sam: Markets are noisy, though. Is there evidence beyond an occasional mismatch between a profit leader and a capability leader?
Alex: In the tool-use accounts, cumulative-return rankings had twenty-one pairwise reversals as the evaluation horizon expanded. The corresponding tool-capability rankings had zero. The other mechanism comparisons also had more reversals in returns than in capability scores.
Sam: Stable rankings could still come from a scoring system that consistently favors particular behavior. How are those capability scores produced?
Alex: They mix programmatic measurements with language-model judges. Tool-call validity and executed rule compliance can be checked from records. Qualities such as evidence faithfulness, memory usefulness, and the adequacy of rule explanations involve judge ratings.
Sam: So the scores are diagnostic measurements, not an independent ground truth about intelligence. What did memory reveal?
Alex: Persistent memory means storing experience for use in later decisions. GPT repeatedly retrieved experience but stored only one memory: a broadly applicable rule. Frequent access did not mean its experience store kept developing.
Sam: Would a larger memory store solve that problem, or did the study separate storage from actual reuse?
Alex: Gemini built the largest store and received high ratings for its contents, yet had the weakest memory-use rating. Grok kept a smaller store but used it most effectively. Across the evaluated models, memory reduced tail loss—the average loss in the worst-return periods—while its effect on total return varied.
Sam: And collaboration could have the same trap: installing several agents doesn’t demonstrate that they work together.
Alex: The authors measured participation across roles and whether specialist evidence reached the manager’s final decision. GPT’s workflow nearly collapsed to a single role, yet its returns improved over its reference account. Qwen integrated substantial evidence but earned less, while every evaluated collaborative configuration increased downside risk.
Sam: Before treating this as a recipe for evaluating all agents, what is the main boundary on the findings?
Alex: This is one market window, not evidence across many market regimes or other environments. Shared conditions make comparisons useful, but each account develops its own history. Also, not every model has reported scores for every configuration, and some diagnostics depend on model judges.
Sam: Your reading, then, is stronger evidence for separating outcomes from mechanisms than for a lasting model ranking?
Alex: That’s my assessment. The matched comparisons and traces show why returns alone miss consequential differences here. They don’t establish permanent capability rankings or guarantee that these additions will help elsewhere.
Sam: Who should read the full document, and where should they start?
Alex: Researchers building persistent agents or designing agent evaluations should read it. Start with the capability-specific evaluation dimensions, then the rule-following analysis. That comparison makes the separation between profits, compliance, and auditable reasoning especially concrete. For everyone else: a successful outcome does not tell you which capability produced it.
Sam: Look at the process before trusting the score.