Martin Klissarov, Jonathan Cook, Diego Antognini, Hao Sun, Jingling Li, Natasha Jaques, Claudiu Musat, Edward Grefenstette
9 min
Abstract
Adapting one's thought process based on corrective feedback is an essential ability in human learning, particularly in collaborative settings. In contrast, the current large language model training paradigm relies heavily on modeling vast, static corpora. While effective for knowledge acquisition, it overlooks the interactive feedback loops essential for models to adapt dynamically to their context. In this work, we propose a framework that treats this interactive in-context learning ability not as an emergent property, but as a distinct, trainable skill. We introduce a scalable method that transforms single-turn verifiable tasks into multi-turn didactic interactions driven by information asymmetry. We first show that current flagship models struggle to integrate corrective feedback on hard reasoning tasks. We then demonstrate that models trained with our approach dramatically improve the ability to interactively learn from language feedback. More specifically, the multi-turn performance of a smaller model nearly reaches that of a model an order of magnitude larger. We also observe robust out-of-distribution generalization: interactive training on math problems transfers to diverse domains like coding, puzzles and maze navigation. Our qualitative analysis suggests that this improvement is due to an enhanced in-context plasticity. Finally, we show that this paradigm offers a unified path to self-improvement. By training the model to predict the teacher's critiques, effectively modeling the feedback environment, we convert this external signal into an internal capability, allowing the model to self-correct even without a teacher.
Sam: The teacher, which is the same model but with secret knowledge like the right answer, spots mistakes and gives plain-language nudges, such as "check your addition there." The student then tries again, and over many such practice chats, the model learns to adapt its thinking turn by turn.
Alex: So the reward is sparse—just a +1 if correct at the end, zero otherwise. That pushes it to really listen and adjust.
Alex: Does that explain why one model in the study nailed the "be concise" request after a long explanation, while another just said "got it" but kept going wordy?
Sam: Yes. In the pencil-and-eraser problem, one top model summarized neatly on the second try, matching the logic without fluff. The other acknowledged the request but repeated the full text, missing the point. The trained models gain what's called in-context plasticity—they change their approach mid-chat based on feedback, like editing an essay after a teacher's note, rather than sticking to the first draft.
Alex: Huh. So even after training just on math chats, it helps with other stuff too?
Sam: The paper suggests yes. A smaller model like Gemini 2.5 Flash, after this math-focused training, nearly matches the larger Gemini 2.5 Pro on tough math over turns—and shows meaningful gains on coding, puzzles, even mazes.
Sam: They test this at use-time in three ways: with a teacher, on general multi-turn games, and self-improvement where the model predicts its own critiques to fix errors alone.
Alex: Right—like internalizing the teacher's voice so it can coach itself. That sidesteps needing a live helper every time.
Alex: But only for checkable problems, like math with a right answer?
Sam: Correct. It relies on tasks you can verify, so open-ended ones remain untested.
Alex: So for problems with clear right answers, this feedback training helps a lot. But how do the top models do out of the box, before any special practice?
Alex: Do bigger ones naturally get better at using hints over several back-and-forths?
Sam: The study first checks that without extra training. They test four tough reasoning jobs—math, puzzles, coding contests, and tricky language logic—using models like Gemini 2.5 Pro, its smaller cousin Flash, and others.
Sam: Bigger models improve a bit more with feedback turns. But all leave room to grow; success rates climb slowly, showing they don't fully use the hints to rethink deeply.
Alex: Huh. So even the best struggle to build on feedback across turns. That sets up why the training matters.
Alex: And when they add this practice—comparing it to simpler ways—what stands out?
Sam: They pit it against two basics: one trains by just copying full answers to problems, like memorizing homework solutions. Another gives rewards per single step if verifiable.
Sam: The new method wins, especially over many turns—it keeps improving while others stall.
Alex: Right—like practicing full conversations teaches adjustment better than isolated fixes.
Alex: What about those thinking models that already pause and check themselves?
Sam: They test on Gemini 2.5 Flash, training it just on harder math chats. On a separate tough math set, the practiced Flash nearly matches the bigger Pro.
Sam: Trained only on math, it boosts performance on unrelated puzzles, coding contests, and logic tasks versus no-transfer baselines. Behaviorally, baselines ignore precise fixes and repeat errors; the trained one rethinks using feedback traces.
Alex: So the practice makes it actually update its plan mid-chat, not just nod.
Alex: And they check the teacher doesn't cheat by leaking answers?
Sam: Yes—a check flags just 0.3 percent possible leaks on average, low noise.
Sam: This verifies real learning from hints. Plus, they train models to guess teacher critiques internally, so at use-time, it self-corrects without help.
Alex: But to see it in action, can you walk through one of those math examples where the trained model actually changes course?
Alex: Like, how does it rethink instead of repeating?
Sam: Consider a tough math problem solving a special equation that describes waves or heat spreading over time and space. The model first guesses wrong, assuming the solution fades to zero far away.
Sam: The teacher hints: "Don't assume it goes to zero—try non-zero limits." The untrained model repeats its error, ignoring the nudge. But the trained one rethinks, tests new forms like approaching 1 plus a square-root term, and gets it right by the end.
Alex: So it builds on the hint to derive a fresh solution, not just tweak the old one.
Alex: And that skill carries to non-math stuff, like games?
Sam: The paper tests this. After math-only practice, the smaller model beats baselines on varied back-and-forth tasks—like navigating mazes or grouping puzzle words.
Sam: No teacher involved; it just adapts over turns in those environments.
Alex: Huh. Training on math chats sharpens general chat skills.
Alex: Now, that self-coaching—how does it work without a real teacher?
Sam: They train the student to predict what feedback the teacher would give, based on past chats—like learning a teacher's grading style so you can spot your own mistakes.
Sam: At use-time, the model critiques its own tries in a loop, refining solo. On math benchmarks over 10 turns, this self-version outperforms even ones using a live teacher.
Alex: So wrapping this up, it seems like a solid way to make chatbots actually listen and adjust in real talks.
Alex: But what are the main limits here, and where does the field go next?
Sam: The approach works well on problems with clear right answers, like math, where you can check success easily. It doesn't yet handle open-ended tasks without those checks, and the paper notes potential risks like the model flattering the user too much instead of being honest—called sycophancy.
Sam: Future steps could include building lesson plans based on repeated mistakes, similar to how teachers design classes step by step. Overall, it's a meaningful advance in making models adapt during chats without needing fancy prompts each time.
Alex: Right, and that self-critique piece turns feedback into something the model carries alone, like an inner editor.
Alex: Scalable for any model with existing data sets—practical without huge new training.
Sam: Exactly. Smaller models close the gap with giants on reasoning after this fine-tuning, and skills transfer to coding or games.
Sam: At use-time, tests confirm gains in teacher-guided chats, general multi-turn tasks, and solo self-improvement.
Alex: So the big takeaway: training on feedback loops boosts real-time learning, rivals size advantages, but stays cautious on risks and untested areas.
Alex: A step toward AIs that evolve mid-conversation, grounded in verifiable evidence. Thanks for listening to ResearchPod.