Adapting one's thought process based on corrective feedback is an essential ability in human learning, particularly in collaborative settings. In contrast, the current large language model training paradigm relies heavily on modeling vast, static corpora. While effective for knowledge acquisition, it overlooks the interactive feedback loops essential for models to adapt dynamically to their context. In this work, we propose a framework that treats this interactive in-context learning ability not as an emergent property, but as a distinct, trainable skill. We introduce a scalable method that transforms single-turn verifiable tasks into multi-turn didactic interactions driven by information asymmetry. We first show that current flagship models struggle to integrate corrective feedback on hard reasoning tasks. We then demonstrate that models trained with our approach dramatically improve the ability to interactively learn from language feedback. More specifically, the multi-turn performance of a smaller model nearly reaches that of a model an order of magnitude larger. We also observe robust out-of-distribution generalization: interactive training on math problems transfers to diverse domains like coding, puzzles and maze navigation. Our qualitative analysis suggests that this improvement is due to an enhanced in-context plasticity. Finally, we show that this paradigm offers a unified path to self-improvement. By training the model to predict the teacher's critiques, effectively modeling the feedback environment, we convert this external signal into an internal capability, allowing the model to self-correct even without a teacher.
Alex: Welcome to another episode of ResearchPod. Today, let's think about something everyday with AI chatbots—they pull facts from huge piles of text they've been trained on, like reciting a recipe from memory.
Alex: But when you're chatting and you give them feedback, say to keep it short, they often nod along but keep rambling on. Why can't they adjust on the fly, like a human tweaking their explanation mid-conversation?
Sam: That's a key gap. These systems, called large language models, are built by feeding them massive amounts of written text so they can predict what comes next in a sentence. They shine at static tasks, like answering trivia from books.
Sam: Yet in a back-and-forth talk, when you point out an error or ask to simplify—like solving a math puzzle where they mess up the steps—they rarely fix their thinking based on your words. The research shows top models struggle here, especially on tough reasoning problems.
Alex: Right, so it's not that they lack the basic knowledge. It's that they freeze when feedback comes in conversation form.
Alex: Take that pencil-and-eraser example from the study—where subtracting costs shows a pencil is 29 cents. The model spits out a wall of text explaining it, and even if told to be concise, it repeats the long version.
Sam: Exactly. The model sees the hint but doesn't shorten its response or rethink efficiently. This points to a deeper issue: without special training, they treat each turn separately, not building or adapting their reasoning across messages.
Sam: The paper suggests the fix is training them specifically to learn interactively—like turning solo math problems into teacher-student chats. Here, a teacher gives hints without spoiling the answer, using extra info it knows, and the student practices adjusting over several back-and-forths.
Alex: Huh. So it's about making adaptation a skill they practice, not something that just happens.
Alex: That could change how we chat with AI.
Sam: It could. The evidence shows smaller models, after this training, handle hard math nearly as well as much bigger ones over multiple turns.
Alex: Okay, so they're practicing these back-and-forths during training. But how exactly do they make the model better at using those hints?
Alex: Like, what's the training trick that lets a smaller AI catch up to a bigger one?
Sam: They use a training method where the model gets rewards only for getting things right in the end—like giving a sticker to a kid only after they solve the full puzzle correctly, not for each step. This encourages the whole conversation to build toward success. Researchers call this setup reinforcement learning with language feedback, or RL²F for short.
Sam: The teacher, which is the same model but with secret knowledge like the right answer, spots mistakes and gives plain-language nudges, such as "check your addition there." The student then tries again, and over many such practice chats, the model learns to adapt its thinking turn by turn.
Alex: So the reward is sparse—just a +1 if correct at the end, zero otherwise. That pushes it to really listen and adjust.
Alex: Does that explain why one model in the study nailed the "be concise" request after a long explanation, while another just said "got it" but kept going wordy?
Sam: Yes. In the pencil-and-eraser problem, one top model summarized neatly on the second try, matching the logic without fluff. The other acknowledged the request but repeated the full text, missing the point. The trained models gain what's called in-context plasticity—they change their approach mid-chat based on feedback, like editing an essay after a teacher's note, rather than sticking to the first draft.
Alex: Huh. So even after training just on math chats, it helps with other stuff too?
Sam: The paper suggests yes. A smaller model like Gemini 2.5 Flash, after this math-focused training, nearly matches the larger Gemini 2.5 Pro on tough math over turns—and shows meaningful gains on coding, puzzles, even mazes.
Sam: They test this at use-time in three ways: with a teacher, on general multi-turn games, and self-improvement where the model predicts its own critiques to fix errors alone.
Alex: Right—like internalizing the teacher's voice so it can coach itself. That sidesteps needing a live helper every time.
Alex: But only for checkable problems, like math with a right answer?
Sam: Correct. It relies on tasks you can verify, so open-ended ones remain untested.
Alex: So for problems with clear right answers, this feedback training helps a lot. But how do the top models do out of the box, before any special practice?
Alex: Do bigger ones naturally get better at using hints over several back-and-forths?
Sam: The study first checks that without extra training. They test four tough reasoning jobs—math, puzzles, coding contests, and tricky language logic—using models like Gemini 2.5 Pro, its smaller cousin Flash, and others.
Sam: Bigger models improve a bit more with feedback turns. But all leave room to grow; success rates climb slowly, showing they don't fully use the hints to rethink deeply.
Alex: Huh. So even the best struggle to build on feedback across turns. That sets up why the training matters.
Alex: And when they add this practice—comparing it to simpler ways—what stands out?
Sam: They pit it against two basics: one trains by just copying full answers to problems, like memorizing homework solutions. Another gives rewards per single step if verifiable.
Sam: The new method wins, especially over many turns—it keeps improving while others stall.
Alex: Right—like practicing full conversations teaches adjustment better than isolated fixes.
Alex: What about those thinking models that already pause and check themselves?
Sam: They test on Gemini 2.5 Flash, training it just on harder math chats. On a separate tough math set, the practiced Flash nearly matches the bigger Pro.
Sam: Trained only on math, it boosts performance on unrelated puzzles, coding contests, and logic tasks versus no-transfer baselines. Behaviorally, baselines ignore precise fixes and repeat errors; the trained one rethinks using feedback traces.
Alex: So the practice makes it actually update its plan mid-chat, not just nod.
Alex: And they check the teacher doesn't cheat by leaking answers?
Sam: Yes—a check flags just 0.3 percent possible leaks on average, low noise.
Sam: This verifies real learning from hints. Plus, they train models to guess teacher critiques internally, so at use-time, it self-corrects without help.
Alex: But to see it in action, can you walk through one of those math examples where the trained model actually changes course?
Alex: Like, how does it rethink instead of repeating?
Sam: Consider a tough math problem solving a special equation that describes waves or heat spreading over time and space. The model first guesses wrong, assuming the solution fades to zero far away.
Sam: The teacher hints: "Don't assume it goes to zero—try non-zero limits." The untrained model repeats its error, ignoring the nudge. But the trained one rethinks, tests new forms like approaching 1 plus a square-root term, and gets it right by the end.
Alex: So it builds on the hint to derive a fresh solution, not just tweak the old one.
Alex: And that skill carries to non-math stuff, like games?
Sam: The paper tests this. After math-only practice, the smaller model beats baselines on varied back-and-forth tasks—like navigating mazes or grouping puzzle words.
Sam: No teacher involved; it just adapts over turns in those environments.
Alex: Huh. Training on math chats sharpens general chat skills.
Alex: Now, that self-coaching—how does it work without a real teacher?
Sam: They train the student to predict what feedback the teacher would give, based on past chats—like learning a teacher's grading style so you can spot your own mistakes.
Sam: At use-time, the model critiques its own tries in a loop, refining solo. On math benchmarks over 10 turns, this self-version outperforms even ones using a live teacher.
Alex: So wrapping this up, it seems like a solid way to make chatbots actually listen and adjust in real talks.
Alex: But what are the main limits here, and where does the field go next?
Sam: The approach works well on problems with clear right answers, like math, where you can check success easily. It doesn't yet handle open-ended tasks without those checks, and the paper notes potential risks like the model flattering the user too much instead of being honest—called sycophancy.
Sam: Future steps could include building lesson plans based on repeated mistakes, similar to how teachers design classes step by step. Overall, it's a meaningful advance in making models adapt during chats without needing fancy prompts each time.
Alex: Right, and that self-critique piece turns feedback into something the model carries alone, like an inner editor.
Alex: Scalable for any model with existing data sets—practical without huge new training.
Sam: Exactly. Smaller models close the gap with giants on reasoning after this fine-tuning, and skills transfer to coding or games.
Sam: At use-time, tests confirm gains in teacher-guided chats, general multi-turn tasks, and solo self-improvement.
Alex: So the big takeaway: training on feedback loops boosts real-time learning, rivals size advantages, but stays cautious on risks and untested areas.
Alex: A step toward AIs that evolve mid-conversation, grounded in verifiable evidence. Thanks for listening to ResearchPod.