Haoyu Zhao, Zafeirios Fountas, Jun Wang
8 min
Abstract
Humans continuously acquire knowledge from carefully crafted materials such as books and tutorial videos, carrying that understanding into future tasks. Millions of such instructional videos are uploaded online every day, yet LLM agents remain unable to exploit this ever-expanding wealth of human teaching. Their capabilities are frozen after training, while parameter updates at scale remain prohibitively expensive. This raises the central question: can LLM agents learn from this vast instructional resource without parameter updates? We introduce Tutorial Agentic Learning (TAL), a nonparametric continual-learning paradigm that consolidates tutorial videos into a persistent, structured tutorialbook memory through a two-stage adversarial process that harvests hard, video-grounded questions and uses them to refine the memory. At inference time, TAL retrieves through a Table-of-Contents (TOC) tree, tying retrieval directly to the model's reasoning rather than to embedding similarity. To study this setting, we introduce VidLearn, a multimodal QA benchmark built from 72 chess tutorial videos spanning 12 openings. On VidLearn, TAL outperforms all evaluated baselines and improves monotonically as more videos are consumed, achieving up to 113% and 91% relative gains over the corresponding vanilla MLLMs on the training and test splits, respectively. In actual gameplay, TAL reaches a stronger opening-phase position than its vanilla counterpart in 69% of head-to-head games. Evaluation in a second domain further demonstrates the generality of TAL. Together, these results highlight instructional videos as a scalable resource for parameter-update-free agent continual learning.
Sam: So the filter is looking for instructional knowledge, not simply whether the model can identify something on the screen. Where does difficulty enter?
Alex: The answer model tries each question without external help. Questions it fails become the question bank. That targets gaps in a frozen model, meaning its learned weights remain unchanged throughout.
Sam: Give me the concrete connection between the video and the resulting knowledge. What would be lost by just keeping subtitles?
Alex: In the chess tutorials, the frame supplies a board position while the lecturer explains it. Their alignment connects the explanation to particular pieces, squares, and spatial relationships. The study tests this directly: building memory with frames and subtitles beats building it from subtitles alone.
Sam: Once the system has found those gaps, how does it turn them into something more useful than saved question-and-answer pairs?
Alex: It writes explanatory chapters, then answers the question bank using those chapters. Remaining mistakes guide targeted rewrites. An edit is accepted only if it improves overall accuracy on that bank; otherwise, the previous version stays.
Sam: That sounds like a useful safeguard, but it only protects the score being checked. It cannot guarantee better answers to unseen questions.
Alex: That’s the boundary. The acceptance rule guarantees improvement on the scoring bank when edits are accepted. It does not guarantee generalization, or that every previously correct answer stays correct. Those claims need separate evaluation.
Sam: Before we get to that evaluation, how does the model find the relevant chapter when a new question arrives?
Alex: It reads a hierarchical table of contents, including chapter titles and summaries, then selects chapters to open. Ordinary similarity-based retrieval matches a query to nearby text representations. Here, the model reasons about which chapter should contain useful knowledge.
Sam: That makes retrieval part of solving the question, rather than an independent matching step. What did the authors use to test whether it helps?
Alex: They built VidLearn from 72 chess tutorial videos covering classical openings. Some videos supply the learning material; others are held out. At evaluation, the model gets a single frame, a written question, and multiple-choice options—not the full tutorial.
Sam: Held-out videos are useful, but how much test material are we talking about?
Alex: The chess test contains 110 questions, with manually verified annotations. So this is a focused benchmark, not a broad survey of agent competence. The training-question scores measure absorption of targeted gaps; the held-out scores are the stronger evidence for transfer.
Sam: And against serious alternatives, rather than just a model answering without help, how large is the gain?
Alex: In the matched comparison, TAL reaches about 72 percent held-out accuracy with the full tutorial collection. The strongest competing baseline reaches about 65 percent. Those methods share the same answer model, and results are averaged across repeated runs.
Sam: That is a clearer comparison than a relative improvement over an initially weak model. Do the component tests support the proposed explanation?
Alex: They do. Keeping harvested questions as case memory helps, and keeping an unrefined tutorialbook helps. Combining question harvesting with memory refinement does better than either alone. A separate retrieval test also favors the table of contents over embedding similarity for finding chapters associated with the correct opening.
Sam: Chess has unusually clear structure. Does the method survive a move into less board-like instruction?
Alex: They also test Photoshop tutorials on several features. With the full learning collection, held-out accuracy reaches about 50 percent. The strongest baseline reaches about 39 percent. That supports transfer beyond chess, though it remains another tutorial-question task.
Sam: The abstract also emphasizes continual improvement as more videos arrive. Does every reported curve actually rise without interruption?
Alex: No. The overall pattern favors larger tutorial collections, but some reported curves have dips, including the software held-out results. I would read the evidence as encouraging scaling across these collections, not a guarantee that each additional batch improves every score.
Sam: And what about actual chess decisions? Answering questions about openings is not the same as choosing moves against another player.
Alex: They compare the same base model with and without tutorial memory, starting from fixed opening positions and swapping colors. Stockfish, a chess engine, evaluates the resulting positions. At the three-move cutoff, TAL has the favorable-position score in about 69 percent of games, counting draws as half.
Sam: That is an opening-position result, then, not a full-game win rate. The distinction matters if someone hears that as general chess strength.
Alex: It does. TAL also makes moves closer to Stockfish’s preferred moves across the tested horizons. But the gameplay evidence is short-horizon and opening-specific. Together with the modest held-out question sets, that limits how far I’d extend the continual-learning claim.
Sam: Given that boundary, who should read the full document, and where should they start?
Alex: Researchers building agent memory or learning from multimodal instruction should read it. Start with Two-Stage Adversarial Consolidation, then the matched retrieval-and-memory comparison. Read the ablations next: they test whether refinement, structured retrieval, and visual evidence each contribute.
Sam: And for researchers who mainly need the idea, rather than an implementation?
Alex: Remember that the model’s weights stayed fixed, but its usable knowledge did not. Better external notes can make a frozen model more capable. This study gives that idea a concrete mechanism and bounded experimental support—not a demonstration of unrestricted lifelong learning.
Sam: Keep the notes, and keep the scope in view.