It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.
Alex: An agent’s notes can be trained, not just its answers. This study finds that rewarding useful notes improves later attempts beyond training the agent only to act.
Sam: That sounds relevant to anyone building an agent that retries after failure. Is the contribution a better memory system, or a different way to train?
Alex: A different way to train, from Doyoung Kim and colleagues at New York University and KAIST AI. Their method paper is an arXiv preprint called “Experience-Handoff Reinforcement Learning for Recursive Self-Improvement.”
Sam: If I already give my agent a place to record mistakes, what would this change?
Alex: You would train it to write notes that help, rather than assume that skill comes with better task performance. We’ll unpack how the authors reward those notes, what their comparisons show, and why the self-improvement claim needs a narrow reading.
Sam: Start with the gap. Agents already receive feedback from tools and failed tests, so what information is being lost?
Alex: In outcome-only reinforcement learning, training behavior using rewards, a whole attempt is scored with a number. The attempt may expose error logs, tool outputs, or environmental constraints. But those details do not become a written record that the next attempt can read.
Sam: The other option is to preserve that record explicitly. The agent reflects on what happened and leaves instructions for its next try.
Alex: That is what a harness can do: fixed scaffolding controlling model calls and carried context. The authors use Agentic Context Engineering, shortened to ACE. Its playbook is a short list of task-specific tips passed to the next attempt.
Sam: So reflection and playbooks already existed. What was missing that makes this more than another way to arrange prompts?
Alex: The authors keep the scaffolding fixed and train the model inside it. Previous harnesses preserve experience, but their frozen models do not learn to write better handoffs from the effects on later attempts. Here, the same model learns to act, reflect, and edit the playbook.
And the title’s “recursive self-improvement” means improving those later attempts? It is not claiming that the deployed model rewrites its own weights.
Alex: Correct. During testing, the weights and harness stay fixed. The model changes only the context its future attempts inherit. Each actor call starts a fresh environment episode, and only the task’s playbook carries across attempts.
Sam: Can we make that concrete? I want to know what useful experience looks like, rather than just a plausible-sounding reflection.
Alex: The paper gives an AppWorld task involving a simulated Venmo account. The agent must remind roommates about old, unresolved payment requests. Its initial attempt fails because it never identifies the roommates and reminds too many people.
Sam: So it needs to repair how it identifies the relevant requests, not merely tell itself to be more careful.
Alex: Its playbook gains a tip to verify roommate identities using supervisor or phone-app data. A later tip says to calculate the pending time using the request’s creation date and a verified current date. Eventually, the actor finds the missing information and sends the correct reminders.
Sam: That example shows useful advice. But how does training distinguish useful advice from advice that just sounds sensible?
Alex: By trying it. The curator proposes an edit, the system applies it to a copy of the playbook, and a fresh actor attempt uses that candidate. The task’s verifier scores the attempt, giving the edit a reward based on what happened downstream.
Sam: So there is no separate model judging whether the note reads well. The environment judges whether the next attempt works.
Alex: That is the central idea. The reflection receives the average score of the candidate playbooks derived from it. The actor receives its own task reward, while invalid edits and edits that change nothing receive no reward.
Sam: But a good next attempt could rescue a poor note. How do they avoid assigning every part of the process the same credit?
Alex: They sample alternative reflections of the same attempt, then alternative edits of the same reflection. Each role is compared within its own group. A reflection can therefore score well even when the failed attempt it describes scored badly.
Sam: They are training both task execution and the handoff, while making the handoff comparisons less dependent on which failure happened first.
Alex: That is the design of ACE-RL, their reinforcement-learning method for this harness. They also reuse the candidate-testing attempts to train the actor. Those tests are not simply extra actor attempts added beyond the baseline’s rollout budget.
Sam: What did they test it against? A weak untrained agent would not tell us whether training the notes actually adds value.
Alex: They test across four benchmarks, spanning a deduction game, programming, app interaction, and software issue resolution. Comparisons include actor-only training, that trained actor placed inside the same harness, and joint training through the harness using a learned reward predictor.
Sam: The trained actor inside the same harness seems the crucial practical comparison. What changes when the model has actually learned to use that setup?
Alex: On the software issue benchmark, MiniSWEBench, ACE-RL reaches about 34 percent held-out test reward, against about 26 percent for the actor-only model inside the harness. That comparison uses the same number of actor attempts and matched training-update compute. ACE-RL also leads on the other tested benchmarks.
Sam: A higher score could still come from a better actor, not better notes. Did they hold the trained actor fixed and remove the handoff?
Alex: They did. They repeated evaluation with the same ACE-RL model but discarded every playbook edit. Each attempt then read an empty playbook, separating the contribution of carried notes from the capabilities of that trained actor.
Sam: That is closer to the mechanism claim. How much difference did preserving the notes make?
Alex: On AppWorld, applying edits adds roughly ten percentage points at an early retry. Within the training horizon, edits help across all the benchmarks, though the programming result is near its ceiling. For me, this control is stronger evidence for useful handoff than the overall ranking alone.
Sam: What happens with longer chains of retries? Notes can accumulate errors as well as useful discoveries.
Alex: Training resets each task’s playbook every three encounters, but evaluation extends to eight attempts. ACE-RL stays ahead of independent attempts from the actor-only baseline. Yet its own notes are not always beneficial: on Mastermind, their advantage fades, and discarding edits eventually does slightly better.
Sam: So the overall lead lasts longer than training, but that does not mean the memory keeps improving indefinitely. What else limits confidence?
Alex: Every method is trained only once. The reported uncertainty is across held-out tasks, not repeated training runs, so it does not establish reliability across training seeds. Also, these are sandboxed tasks with verifiers and task-local notes, not open-ended agents accumulating experience across unrelated work.
Sam: Who should read the full paper, and where should they start?
Alex: Researchers training agents or designing memory systems should start with the ACE-RL method section, especially the separate comparison groups. Then read “Where Does the Gain Come From?” for the edits-discarded control. For everyone else, remember this: writing useful experience for the next attempt is itself a behavior worth training.
Sam: Keep that distinction in mind when your agent tries again.