Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with retries permitted on incorrect submissions, which are paid for. Each instance is scored based on the share of this feedback budget consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization. We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. Persistence in the harness generally does not improve memorization unless provided with explicit expert guidance. Most harness-based methods still learn less than a floor that simply caches each task's demonstrations, across all models we tested. No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.
Alex: An AI agent can keep notes without getting better at its job. In a new benchmark, stored notes usually helped less than simply saving examples, while training model weights improved unfamiliar-task performance.
Sam: That challenges the idea that giving an agent memory makes it a continual learner. What kind of evidence is this?
Alex: It’s Continual-ARC, a benchmark from Daniel Lenton, Yusha Arif, and Haris Mahmood at Unify. The paper is a preprint under review. It tests whether experience makes later work easier, rather than just leaving more files behind.
Sam: For researchers building agents, that distinction affects what they should store, train, and measure. Let’s unpack how the benchmark separates remembering from improving, what actually worked, and how far these results travel.
Alex: Start with the puzzles. The Abstraction and Reasoning Corpus, or ARC, asks a system to infer a rule from example input and output grids. It then applies that rule to another grid. The tasks need basic priors such as counting and objectness, rather than specialist domain knowledge.
Sam: Ordinary ARC already tests learning from examples. What was missing that required another benchmark?
Alex: Ordinary ARC measures skill acquisition within an episode. It doesn’t ask whether that skill survives until the next similar job. Existing agent benchmarks often use coding or office work, where models already know a great deal, and instances usually appear once or return unchanged.
Sam: If an identical instance comes back, a saved answer might look like a learned skill. So the crucial change is bringing back the rule without bringing back the same question.
Alex: That’s the benchmark’s central move. A task returns after a dormant gap, but with a fresh input generated for that rule. New tasks also arrive throughout the stream. The main experiments draw on twenty-five tasks from the original ARC training set.
Sam: What does the learner actually see? Without knowing which task has returned, retrieving the right memory could become a different problem entirely.
Alex: It receives a task identifier and a fresh input grid. So it knows which task it faces, but not its position in the stream. The main stream contains two hundred instances, with tasks appearing in bursts separated by gaps.
Sam: And what counts as doing better? Solving more puzzles is useful, but that could hide repeated spending on the same lessons.
Alex: The authors charge for feedback. Requesting a demonstration costs the same as submitting a wrong answer; a correct answer ends the instance. Failure costs the full feedback budget. Lower spending is better, and the authors report solve rates alongside it.
Sam: So an agent benefits by remembering demonstrations it already bought. It benefits further if it learns a reusable rule and stops needing those demonstrations.
Alex: Consider their simple demonstration cache. It stores each task’s purchased examples and replays them free when that task returns. Their engineered program library goes further: it asks for a Python solution, checks it against demonstrations, then saves and reuses it.
Sam: Those are useful references. But a powerful model might spend little from the start, without learning anything during the stream. How do they separate initial ability from learning?
Alex: Every learner gets a reset comparison: the same system facing the same stream, with its state wiped before each instance. The authors measure savings relative to that reset. For learning during the stream, they also account for the unavoidable cost of first encountering each task.
Sam: That handles improvement along the way. How do they distinguish retaining a particular skill from becoming a better solver overall?
Alex: After the stream, they freeze the learner’s accumulated state and run separate tests. One uses fresh inputs from familiar tasks. The other uses a hundred evaluation tasks the stream never showed. Each test instance starts from the same end-of-stream state, so test experience cannot accumulate.
Sam: Their word “memorization” therefore means retaining a task’s rule, not memorizing its old answer. And “generalization” means improvement on tasks the learner hasn’t encountered.
Alex: Yes, and that distinction drives the findings. For the stronger GLM model, simply caching demonstrations saved forty-four percent of the familiar-task feedback budget relative to its stateless comparison. That’s a substantial target for a more elaborate memory system to beat.
Sam: What did those more elaborate systems contain? “Memory” could mean anything from a transcript to a searchable collection of programs.
Alex: The authors tested agent harnesses: the prompts, tools, and storage surrounding a model. Their main harness opened a fresh conversation for every instance, carrying forward only written state. Without special guidance, persistence generally helped little, and most harness methods learned less than demonstration caching.
Sam: A system can’t use a memory it never retrieves. Did they inspect whether the problem was learning the wrong thing, storing nothing, or failing to bring useful material back?
Alex: They found all those failure patterns. GLM stored verbal rules, but its harness exposed only a small selection of entries, chosen before the task was known. Other inspected configurations failed to preserve lasting entries, or kept general advice that didn’t reduce spending.
Sam: That sounds partly like a storage-and-retrieval failure, rather than proof that external memory cannot work. Did directing the agents toward useful behavior change the result?
Alex: It did. Expert guidance explicitly told agents to write Python programs, verify them against demonstrations, save them by task, and reuse them. With that guidance, GLM saved seventy-four percent on familiar tasks relative to its reset. The harness could support effective retention when directed appropriately.
Sam: But a stored program for a familiar rule doesn’t necessarily help with a different rule. Did any memory-based method improve on unfamiliar tasks?
Alex: None improved measurably in these experiments. The authors note that familiar and held-out tasks share primitive building blocks, so reusable abstractions were possible in principle. But even notes and programs written by the stronger teacher did not produce measurable transfer for the student.
Sam: What changed when they trained the model itself, rather than giving it better notes?
Alex: They used DAgger, which trains on teacher replies at situations the student actually encounters. Between rounds, they updated a small trainable adapter to the Qwen student model. The stronger GLM teacher received expert guidance that the student did not.
Sam: So this wasn’t learning from sparse correct-or-incorrect feedback alone. It had a stronger teacher supplying behavior to imitate. What improvement did that buy?
Alex: On unfamiliar tasks, weight training saved twenty-nine percent of the feedback budget relative to the untrained reset. On familiar tasks, it saved fifty-seven percent. The reported uncertainty intervals excluded no improvement for both results.
Sam: How much should we read into that contrast? The training method receives richer supervision, and the student initially struggled to operate the harness.
Alex: Both qualifications matter. The authors say some gains came from learning to drive the harness, although trained performance also exceeded the simpler interaction loop. Most runs used relatively small models on the easiest ARC track, and the quoted results use a fixed stream seed. This does not establish that weights generally beat memory.
Sam: Given those limits, who should read the full document, and where should they start?
Alex: Researchers building agent memory or continual-learning evaluations should start with the benchmark sections on the post-stream tests and reset comparisons. Then read the results on expert guidance and generalization. Those comparisons separate failure to store useful experience from failure to transfer it.
Sam: And for someone who just needs the main lesson?
Alex: Saving experience is not the same as learning from it. Ask whether later work becomes easier—and whether that improvement survives a genuinely new task.
Sam: That’s a useful test to carry back to the lab.