Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.
Alex: Welcome to another episode of ResearchPod. Sam, what are we looking at today?
Sam: We're looking at a paper called "Chain-of-Experience for Continual LLM Improvement." It asks a deceptively simple question: why do AI language models treat every problem as if they've never seen anything like it before?
Alex: That does seem like a strange design choice. Humans learn from their mistakes. Why don't these models?
Sam: It comes down to how they're built. When a language model is trained, it learns from a massive collection of text. But once that training is done, the model is essentially frozen. Every time you ask it a question, it starts completely fresh — no memory of what worked before, no record of what went wrong.
Alex: So it's like a student who has to retake every exam from scratch, with no notes and no memory of past tests.
Sam: That's a fair way to put it. And this paper proposes a fix. Instead of discarding everything that happens during a conversation, the system keeps a running record — a kind of experience log — of what the model tried, what feedback it received, and what that feedback implied about where it went wrong.
Alex: What does that feedback actually look like in practice?
Sam: In coding tasks, for example, the system runs the model's code through an interpreter. If the code crashes or fails a test, the error message gets fed directly back to the model. So rather than guessing again from scratch, the model sees exactly what broke and why, and uses that to shape its next attempt.
Alex: So it's less like guessing and more like debugging. You try something, see the error, and fix it.
Sam: Precisely. And the paper shows this matters. When models were given access to this kind of iterative feedback loop, they consistently outperformed versions that had to answer without any feedback — and they did so while using fewer computational resources overall.
Alex: Wait — it gets better results and costs less to run?
Sam: That's what the findings suggest. The reason is efficiency. When a model can see what it already tried and why it failed, it stops wasting effort repeating the same dead ends. It narrows in on the right answer faster.
Alex: That makes sense. But I'm curious — what kind of feedback actually helps? Is it always an objective signal like a code error, or can the model critique itself?
Sam: Both, it turns out — and they work best together. The paper explores combining two types of signals. One is objective: a math answer is either right or wrong, or code either runs or it doesn't. The other is the model's own self-reflection — essentially asking it to evaluate its own reasoning before trying again.
Alex: And those two things complement each other?
Sam: They appear to. On competitive math problems, combining both signals reached notably higher accuracy than either signal used alone. What's particularly interesting is that self-generated feedback — the model critiquing itself — accounted for a substantial share of the improvements in certain setups. The model's own internal critique was often well-aligned with what it actually needed to fix.
Alex: Though I'd imagine there's a risk there. What if the model's self-critique is just wrong?
Sam: The authors tested exactly that — deliberately feeding models incorrect feedback signals to see what would happen. Performance dropped on average, as you'd expect. But stronger models showed meaningful resilience. And one technique helped significantly: instead of relying on a single attempt, the system gathered multiple independent responses and went with the answer the majority agreed on.
Alex: So if you ask the same question several different ways and most answers converge, you can filter out the noise from a bad signal.
Sam: That's the idea. And when the researchers dug into thousands of individual cases where the model went from a wrong answer to a correct one, roughly half of those improvements traced directly back to the model interpreting and acting on the feedback it received. The feedback wasn't decoration — it was doing real work.
Alex: So the core insight is that the model doesn't need to be smarter from the start. It needs the chance to practice and course-correct in real time.
Sam: That's a good way to frame it. Though the paper is careful to note the limitations. As the experience log grows longer with each round, the system has more text to process, which increases both the time it takes to respond and the computational load. There's a cost to carrying that history.
Alex: And I'd guess not every model handles that equally well.
Sam: Correct. Models with weaker foundations struggled to generate reliable self-critiques. The technique depends on the model already having enough capability to recognize when its own reasoning has gone astray. A student has to know enough to spot their own mistakes before they can fix them.
Alex: There's also something worth noting about what this approach doesn't do, right? It's not actually rewriting the model.
Sam: That's an important distinction. The improvements here come from how the model uses context during a conversation — not from permanently changing the underlying system. The model is, as you put it, borrowing its past notes rather than rewiring itself. The authors identify permanently incorporating those experiential traces into the model's weights as a meaningful direction for future work.
Alex: So this is a real step forward, but there's still ground to cover. Thanks for walking through it, Sam.
Sam: Thank you, Alex. The core finding is a practical one: giving a model a structured way to learn from its own attempts — even within a single session — leads to more reliable answers, and that's worth taking seriously. Thanks for listening to ResearchPod.