Teng Pan, Zixuan Wang, Yuchen Yan, Qifei Wu, Yao Chen, Yingqi Qu, Ruiqing Zhang, Yongliang Shen
7 min
Abstract
Large language models have made rapid progress in agentic reasoning and tool use, raising expectations for agents that can take over realistic productivity work from humans. However, current agents still struggle to finish such tasks, where success requires following long-horizon professional workflows, understanding heterogeneous file contexts, and producing usable deliverables under fine-grained requirements. A key bottleneck is the lack of scalable post-training data that reflects real work intent, authentic file environments, and deliverable-level evaluation. To close this gap, we introduce Agent-R$^2$, a $\textbf{r}$eal-source data synthesis framework that scales training data for $\textbf{r}$ealistic productivity work. Instead of relying on manually specified workflows or LLM-generated files, Agent-R$^2$ grounds synthesis throughout the pipeline in real occupational seeds, corpus-derived workflow evidence, and authentic web-sourced files. From 10k web-corpus candidates, Agent-R$^2$ distills 4k task instances that pair real multi-format attachments with fine-grained, deliverable-level rubrics. On this data, we train Agent-R$^2$-4B via supervised fine-tuning and rubric-based reinforcement learning, improving both end-to-end productivity performance and file-grounded reasoning. Trained on only 4k instances, Agent-R$^2$-4B surpasses Qwen3-235B-A22B-Thinking-2507 on both GDPval (51.0$\%$ vs. 48.0$\%$) and OfficeQA (14.8$\%$ vs. 13.8$\%$). Further analysis shows that Agent-R$^2$ mitigates shortcut behavior, where agents produce superficially plausible deliverables before sufficiently grounding, constructing, or verifying them, while promoting sustained exploration.
Alex: So the workflow guides the search, but the retrieved evidence also constrains what the assignment can reasonably ask.
Sam: The next stage is designed to make that connection. A synthesis model examines the files and writes instructions plus a rubric—a detailed checklist for judging the finished work. Across the pipeline, they retain about four thousand training tasks, with multiple attachments typically needed for each.
Alex: A checklist could still reward something impossible, or something the files don't support. How do they check that?
Sam: They have Claude Opus solve the task and keep its deliverables as references. A verifier checks those files against the rubric. If a failed criterion is unreasonable or unsupported, they revise it; otherwise, they ask the expert model to revise the deliverable.
Alex: Then how does the smaller agent learn from those tasks?
Sam: First comes supervised fine-tuning: learning to imitate the expert's actions. Then comes reinforcement learning: improving through rewards for the finished result. A judging model scores the deliverable against the rubric and references, including both its content and presentation.
Alex: That makes the reward about the work product, rather than whether the agent followed a prescribed sequence of clicks or commands.
Sam: Yes. They use outcome rewards without explicit penalties for intermediate tool errors. The aim is to let the agent recover from mistakes and still satisfy the assignment.
Alex: Give me a concrete failure this is supposed to change.
Sam: Their materials-testing example asks for a report based on spreadsheet measurements. The base model fails to locate the needed values but writes a report using dummy evidence. Another comparison model silently converts measurements into missing values and produces empty charts.
Alex: Both can leave a file behind, even though the analysis underneath has failed.
Sam: The trained agent also encounters errors. But it changes its parsing approach, calculates the required hardness measurements, recovers from plotting problems, and checks the final plots. That's the behavior the authors want: not flawless execution, but evidence-grounded recovery.
Alex: A case study makes that visible. How much does the aggregate evaluation support it?
Sam: On GDPval, which evaluates professional deliverables against detailed criteria, the trained agent scores about fifty-one percent on its first attempt. The much larger Qwen comparison model scores forty-eight percent. The trained agent also substantially improves over its own starting model, though stronger models remain ahead.
Alex: Could that mostly reflect learning how to package documents in the way the judge likes?
Sam: They test that concern with OfficeQA, a document-question-answering benchmark, using its plain-text Markdown version. Performance rises from roughly two percent for the starting model to roughly fifteen percent after training. That supports transfer beyond visual polish, though the final score also shows considerable room for improvement.
Alex: There's another explanation: the agent simply copies a strong teacher. Does reinforcement learning add anything beyond seeing more demonstrations?
Sam: They test that with a control trained on all the expert demonstrations. It barely improves over their standard demonstration-trained model and remains below the version with reinforcement learning. Separate comparisons also favor real attachments over generated ones, and detailed rubric rewards over rewards without those rubrics.
Alex: What changes in the agent's behavior? Taking more steps alone wouldn't convince me that it understood more.
Sam: Nor should it. Imitating the expert initially increases exploration, but also increases errors and reduces completion. Reinforcement learning then improves completion while reducing errors, without returning to the starting model's shallow inspection.
Alex: So the evidence isn't simply that longer trajectories are better.
Sam: Their additional analysis explicitly rejects that reading. Within each model, longer runs tend to score lower. After reinforcement learning, redundant computation decreases while reading, writing, and checking retain a larger share of the work.
Alex: Where should we put the biggest qualification? The tasks themselves still come through a synthesis model.
Sam: That's the key limitation. In a manual audit, only eighty-five percent of retained tasks were judged feasible from their instructions and attachments. Real files do not automatically make a coherent assignment, and some generated questions only partially match their source workflows.
Alex: And the benchmark scores depend on model judges. Did the ranking survive changing the judge?
Sam: For the headline Qwen comparison, it survived repeated evaluations with both tested judges. They also audited training materials for benchmark overlap and found no matching attachment hashes or recovered source URLs. Those checks strengthen the benchmark result; they don't turn it into evidence of dependable workplace deployment.
Alex: Who should read the full document, and where would you send them first?
Sam: Researchers building agent-training pipelines or deliverable evaluations should read Data Synthesis first. Then examine the comparison separating additional demonstrations from reinforcement learning, and the manual feasibility audit in Data Details. Together, those sections show the mechanism, the strongest control, and the remaining weakness.
Alex: And for someone who just needs the lesson to carry into their own use of agents?
Sam: A finished file is not evidence of finished work. This study suggests that training should reward grounding, construction, and verification together—not merely the appearance of completion.
Alex: Keep that distinction when you delegate.