Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang
7 min
Abstract
It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.
Alex: The paper gives an AppWorld task involving a simulated Venmo account. The agent must remind roommates about old, unresolved payment requests. Its initial attempt fails because it never identifies the roommates and reminds too many people.
Sam: So it needs to repair how it identifies the relevant requests, not merely tell itself to be more careful.
Alex: Its playbook gains a tip to verify roommate identities using supervisor or phone-app data. A later tip says to calculate the pending time using the request’s creation date and a verified current date. Eventually, the actor finds the missing information and sends the correct reminders.
Sam: That example shows useful advice. But how does training distinguish useful advice from advice that just sounds sensible?
Alex: By trying it. The curator proposes an edit, the system applies it to a copy of the playbook, and a fresh actor attempt uses that candidate. The task’s verifier scores the attempt, giving the edit a reward based on what happened downstream.
Sam: So there is no separate model judging whether the note reads well. The environment judges whether the next attempt works.
Alex: That is the central idea. The reflection receives the average score of the candidate playbooks derived from it. The actor receives its own task reward, while invalid edits and edits that change nothing receive no reward.
Sam: But a good next attempt could rescue a poor note. How do they avoid assigning every part of the process the same credit?
Alex: They sample alternative reflections of the same attempt, then alternative edits of the same reflection. Each role is compared within its own group. A reflection can therefore score well even when the failed attempt it describes scored badly.
Sam: They are training both task execution and the handoff, while making the handoff comparisons less dependent on which failure happened first.
Alex: That is the design of ACE-RL, their reinforcement-learning method for this harness. They also reuse the candidate-testing attempts to train the actor. Those tests are not simply extra actor attempts added beyond the baseline’s rollout budget.
Sam: What did they test it against? A weak untrained agent would not tell us whether training the notes actually adds value.
Alex: They test across four benchmarks, spanning a deduction game, programming, app interaction, and software issue resolution. Comparisons include actor-only training, that trained actor placed inside the same harness, and joint training through the harness using a learned reward predictor.
Sam: The trained actor inside the same harness seems the crucial practical comparison. What changes when the model has actually learned to use that setup?
Alex: On the software issue benchmark, MiniSWEBench, ACE-RL reaches about 34 percent held-out test reward, against about 26 percent for the actor-only model inside the harness. That comparison uses the same number of actor attempts and matched training-update compute. ACE-RL also leads on the other tested benchmarks.
Sam: A higher score could still come from a better actor, not better notes. Did they hold the trained actor fixed and remove the handoff?
Alex: They did. They repeated evaluation with the same ACE-RL model but discarded every playbook edit. Each attempt then read an empty playbook, separating the contribution of carried notes from the capabilities of that trained actor.
Sam: That is closer to the mechanism claim. How much difference did preserving the notes make?
Alex: On AppWorld, applying edits adds roughly ten percentage points at an early retry. Within the training horizon, edits help across all the benchmarks, though the programming result is near its ceiling. For me, this control is stronger evidence for useful handoff than the overall ranking alone.
Sam: What happens with longer chains of retries? Notes can accumulate errors as well as useful discoveries.
Alex: Training resets each task’s playbook every three encounters, but evaluation extends to eight attempts. ACE-RL stays ahead of independent attempts from the actor-only baseline. Yet its own notes are not always beneficial: on Mastermind, their advantage fades, and discarding edits eventually does slightly better.
Sam: So the overall lead lasts longer than training, but that does not mean the memory keeps improving indefinitely. What else limits confidence?
Alex: Every method is trained only once. The reported uncertainty is across held-out tasks, not repeated training runs, so it does not establish reliability across training seeds. Also, these are sandboxed tasks with verifiers and task-local notes, not open-ended agents accumulating experience across unrelated work.
Sam: Who should read the full paper, and where should they start?
Alex: Researchers training agents or designing memory systems should start with the ACE-RL method section, especially the separate comparison groups. Then read “Where Does the Gain Come From?” for the edits-discarded control. For everyone else, remember this: writing useful experience for the next attempt is itself a behavior worth training.
Sam: Keep that distinction in mind when your agent tries again.