Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Five frontier LLMs operate along continuous trajectories under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. We evaluate them through both realized outcomes and mechanism-specific diagnostics derived from complete decision traces. Across 30 days of live evaluation, we find a pronounced outcome-capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use. Trace-level diagnostics further expose distinct bottlenecks across capabilities, demonstrating that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability. LiveMACEBench makes this distinction measurable, turning live markets from a performance leaderboard into a diagnostic environment for agent capability
Alex: An AI agent can make money while using its tools badly. In this study, trading returns often failed to reveal how effectively agents used tools, memory, rules, or collaborators.
Sam: If success hides those differences, that matters beyond trading. What kind of evidence are we talking about?
Alex: A benchmark preprint by Jun Zhao, Leiming Fu, and colleagues, called LiveMACE. They followed five language models in live markets for about thirty days. The point was to diagnose agent capabilities, not just rank profits.
Sam: So this is for researchers deciding whether adding memory or extra agents actually improves their system, rather than just its score?
Alex: Yes, and for anyone choosing how to evaluate those systems. We’ll look at how the benchmark separates outcomes from behavior, what its strongest comparisons show, and where the evidence stops.
Sam: Researchers already evaluate agents using intermediate steps. What does this add beyond another detailed trading leaderboard?
Alex: The authors acknowledge earlier process-aware benchmarks, including AgentBoard, which examine more than final success. They also cite live-market evaluations such as Alpha Arena. Their contribution combines those directions: matched agent configurations, persistent accounts, and diagnostics of how specific mechanisms get used.
Sam: The persistent account seems central. A bad decision doesn’t disappear when the next evaluation task starts.
Alex: It stays with the agent. These are paper-trading accounts: simulated trades using live market data. The agents operate in cryptocurrency and American equity markets, and positions carry forward rather than resetting.
Sam: What makes the comparison controlled when the market keeps changing underneath everyone?
Alex: Agents begin decision rounds on a shared schedule, every four hours. They use a common reasoning-and-action setup. For each model, the authors compare a reference configuration with variants adding expanded tools, persistent memory, explicit rules, or collaboration among specialized agents.
Sam: Those additions sound quite different. How do they avoid treating every improvement as evidence for the same general capability?
Alex: By measuring each mechanism separately. They record a decision trace: the model’s responses, tool calls, tool outputs, and actions within a round. Then they ask whether the added mechanism was used effectively, alongside whether the account performed better.
Sam: So there are separate questions: did the system have the mechanism, did it use it well, and did that help?
Alex: That distinction organizes the whole benchmark. Take rule following. The agent receives hard constraints and lower-priority preferences, with an explicit hierarchy. The evaluation checks both whether its actions comply and whether its explanation demonstrates correct, state-grounded rule reasoning.
Sam: Give me the comparison where looking only at the result would lead me astray.
Alex: Under rule guidance, GPT and DeepSeek finished with almost identical returns. But GPT passed the hard-rule checks about seventy percent of the time, versus roughly ninety percent for DeepSeek. Similar profits concealed quite different compliance.
Sam: That makes the practical distinction clear. If I need a constrained agent, matching another system’s profit doesn’t establish that it respects the constraints.
Alex: And compliance alone doesn’t establish good explanations. DeepSeek’s executed decisions often passed, but its written reasoning was less auditable. Gemini performed strongly on both execution and explanations, despite earning less than the return leaders.
Sam: Were the failures mostly agents ignoring instructions, or could the traces locate something more specific?
Alex: A recurring problem was the limit on exposure to a single asset. Some agents incorrectly combined existing holdings, proposed orders, and price changes when estimating their position after a trade. Recognizing the rule was not enough; they also needed an accurate account of their state.
Sam: That’s a more useful diagnosis than saying a model is weak at instruction following. Does the same separation show up with tools?
Alex: Yes. The evaluated tool-use agents generally gathered enough information, but invocation efficiency was their weakest judged dimension. They could reach adequate coverage through redundant interactions. The authors also separated finding an appropriate tool from successfully calling it.
Sam: Because a system could discover the right source and still fail at execution?
Alex: That happened. Qwen was relatively weak at tool discovery but reliable once a tool was available. Grok showed the opposite pattern. GPT was strong at both stages, yet Grok earned the highest return in that configuration.
Sam: Markets are noisy, though. Is there evidence beyond an occasional mismatch between a profit leader and a capability leader?
Alex: In the tool-use accounts, cumulative-return rankings had twenty-one pairwise reversals as the evaluation horizon expanded. The corresponding tool-capability rankings had zero. The other mechanism comparisons also had more reversals in returns than in capability scores.
Sam: Stable rankings could still come from a scoring system that consistently favors particular behavior. How are those capability scores produced?
Alex: They mix programmatic measurements with language-model judges. Tool-call validity and executed rule compliance can be checked from records. Qualities such as evidence faithfulness, memory usefulness, and the adequacy of rule explanations involve judge ratings.
Sam: So the scores are diagnostic measurements, not an independent ground truth about intelligence. What did memory reveal?
Alex: Persistent memory means storing experience for use in later decisions. GPT repeatedly retrieved experience but stored only one memory: a broadly applicable rule. Frequent access did not mean its experience store kept developing.
Sam: Would a larger memory store solve that problem, or did the study separate storage from actual reuse?
Alex: Gemini built the largest store and received high ratings for its contents, yet had the weakest memory-use rating. Grok kept a smaller store but used it most effectively. Across the evaluated models, memory reduced tail loss—the average loss in the worst-return periods—while its effect on total return varied.
Sam: And collaboration could have the same trap: installing several agents doesn’t demonstrate that they work together.
Alex: The authors measured participation across roles and whether specialist evidence reached the manager’s final decision. GPT’s workflow nearly collapsed to a single role, yet its returns improved over its reference account. Qwen integrated substantial evidence but earned less, while every evaluated collaborative configuration increased downside risk.
Sam: Before treating this as a recipe for evaluating all agents, what is the main boundary on the findings?
Alex: This is one market window, not evidence across many market regimes or other environments. Shared conditions make comparisons useful, but each account develops its own history. Also, not every model has reported scores for every configuration, and some diagnostics depend on model judges.
Sam: Your reading, then, is stronger evidence for separating outcomes from mechanisms than for a lasting model ranking?
Alex: That’s my assessment. The matched comparisons and traces show why returns alone miss consequential differences here. They don’t establish permanent capability rankings or guarantee that these additions will help elsewhere.
Sam: Who should read the full document, and where should they start?
Alex: Researchers building persistent agents or designing agent evaluations should read it. Start with the capability-specific evaluation dimensions, then the rule-following analysis. That comparison makes the separation between profits, compliance, and auditable reasoning especially concrete. For everyone else: a successful outcome does not tell you which capability produced it.
Sam: Look at the process before trusting the score.