Haoqin Tu, Yunhao Fang, Yizhong Wang, Cihang Xie, Shen Yan
5 min
Abstract
Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.
Alex: And those two things complement each other?
Sam: They appear to. On competitive math problems, combining both signals reached notably higher accuracy than either signal used alone. What's particularly interesting is that self-generated feedback — the model critiquing itself — accounted for a substantial share of the improvements in certain setups. The model's own internal critique was often well-aligned with what it actually needed to fix.
Alex: Though I'd imagine there's a risk there. What if the model's self-critique is just wrong?
Sam: The authors tested exactly that — deliberately feeding models incorrect feedback signals to see what would happen. Performance dropped on average, as you'd expect. But stronger models showed meaningful resilience. And one technique helped significantly: instead of relying on a single attempt, the system gathered multiple independent responses and went with the answer the majority agreed on.
Alex: So if you ask the same question several different ways and most answers converge, you can filter out the noise from a bad signal.
Sam: That's the idea. And when the researchers dug into thousands of individual cases where the model went from a wrong answer to a correct one, roughly half of those improvements traced directly back to the model interpreting and acting on the feedback it received. The feedback wasn't decoration — it was doing real work.
Alex: So the core insight is that the model doesn't need to be smarter from the start. It needs the chance to practice and course-correct in real time.
Sam: That's a good way to frame it. Though the paper is careful to note the limitations. As the experience log grows longer with each round, the system has more text to process, which increases both the time it takes to respond and the computational load. There's a cost to carrying that history.
Alex: And I'd guess not every model handles that equally well.
Sam: Correct. Models with weaker foundations struggled to generate reliable self-critiques. The technique depends on the model already having enough capability to recognize when its own reasoning has gone astray. A student has to know enough to spot their own mistakes before they can fix them.
Alex: There's also something worth noting about what this approach doesn't do, right? It's not actually rewriting the model.
Sam: That's an important distinction. The improvements here come from how the model uses context during a conversation — not from permanently changing the underlying system. The model is, as you put it, borrowing its past notes rather than rewiring itself. The authors identify permanently incorporating those experiential traces into the model's weights as a meaningful direction for future work.
Alex: So this is a real step forward, but there's still ground to cover. Thanks for walking through it, Sam.
Sam: Thank you, Alex. The core finding is a practical one: giving a model a structured way to learn from its own attempts — even within a single session — leads to more reliable answers, and that's worth taking seriously. Thanks for listening to ResearchPod.