Daniel Lenton, Yusha Arif, Haris Mahmood
7 min
Abstract
Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with retries permitted on incorrect submissions, which are paid for. Each instance is scored based on the share of this feedback budget consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization. We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. Persistence in the harness generally does not improve memorization unless provided with explicit expert guidance. Most harness-based methods still learn less than a floor that simply caches each task's demonstrations, across all models we tested. No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.
Sam: So an agent benefits by remembering demonstrations it already bought. It benefits further if it learns a reusable rule and stops needing those demonstrations.
Alex: Consider their simple demonstration cache. It stores each task’s purchased examples and replays them free when that task returns. Their engineered program library goes further: it asks for a Python solution, checks it against demonstrations, then saves and reuses it.
Sam: Those are useful references. But a powerful model might spend little from the start, without learning anything during the stream. How do they separate initial ability from learning?
Alex: Every learner gets a reset comparison: the same system facing the same stream, with its state wiped before each instance. The authors measure savings relative to that reset. For learning during the stream, they also account for the unavoidable cost of first encountering each task.
Sam: That handles improvement along the way. How do they distinguish retaining a particular skill from becoming a better solver overall?
Alex: After the stream, they freeze the learner’s accumulated state and run separate tests. One uses fresh inputs from familiar tasks. The other uses a hundred evaluation tasks the stream never showed. Each test instance starts from the same end-of-stream state, so test experience cannot accumulate.
Sam: Their word “memorization” therefore means retaining a task’s rule, not memorizing its old answer. And “generalization” means improvement on tasks the learner hasn’t encountered.
Alex: Yes, and that distinction drives the findings. For the stronger GLM model, simply caching demonstrations saved forty-four percent of the familiar-task feedback budget relative to its stateless comparison. That’s a substantial target for a more elaborate memory system to beat.
Sam: What did those more elaborate systems contain? “Memory” could mean anything from a transcript to a searchable collection of programs.
Alex: The authors tested agent harnesses: the prompts, tools, and storage surrounding a model. Their main harness opened a fresh conversation for every instance, carrying forward only written state. Without special guidance, persistence generally helped little, and most harness methods learned less than demonstration caching.
Sam: A system can’t use a memory it never retrieves. Did they inspect whether the problem was learning the wrong thing, storing nothing, or failing to bring useful material back?
Alex: They found all those failure patterns. GLM stored verbal rules, but its harness exposed only a small selection of entries, chosen before the task was known. Other inspected configurations failed to preserve lasting entries, or kept general advice that didn’t reduce spending.
Sam: That sounds partly like a storage-and-retrieval failure, rather than proof that external memory cannot work. Did directing the agents toward useful behavior change the result?
Alex: It did. Expert guidance explicitly told agents to write Python programs, verify them against demonstrations, save them by task, and reuse them. With that guidance, GLM saved seventy-four percent on familiar tasks relative to its reset. The harness could support effective retention when directed appropriately.
Sam: But a stored program for a familiar rule doesn’t necessarily help with a different rule. Did any memory-based method improve on unfamiliar tasks?
Alex: None improved measurably in these experiments. The authors note that familiar and held-out tasks share primitive building blocks, so reusable abstractions were possible in principle. But even notes and programs written by the stronger teacher did not produce measurable transfer for the student.
Sam: What changed when they trained the model itself, rather than giving it better notes?
Alex: They used DAgger, which trains on teacher replies at situations the student actually encounters. Between rounds, they updated a small trainable adapter to the Qwen student model. The stronger GLM teacher received expert guidance that the student did not.
Sam: So this wasn’t learning from sparse correct-or-incorrect feedback alone. It had a stronger teacher supplying behavior to imitate. What improvement did that buy?
Alex: On unfamiliar tasks, weight training saved twenty-nine percent of the feedback budget relative to the untrained reset. On familiar tasks, it saved fifty-seven percent. The reported uncertainty intervals excluded no improvement for both results.
Sam: How much should we read into that contrast? The training method receives richer supervision, and the student initially struggled to operate the harness.
Alex: Both qualifications matter. The authors say some gains came from learning to drive the harness, although trained performance also exceeded the simpler interaction loop. Most runs used relatively small models on the easiest ARC track, and the quoted results use a fixed stream seed. This does not establish that weights generally beat memory.
Sam: Given those limits, who should read the full document, and where should they start?
Alex: Researchers building agent memory or continual-learning evaluations should start with the benchmark sections on the post-stream tests and reset comparisons. Then read the results on expert guidance and generalization. Those comparisons separate failure to store useful experience from failure to transfer it.
Sam: And for someone who just needs the main lesson?
Alex: Saving experience is not the same as learning from it. Ask whether later work becomes easier—and whether that improvement survives a genuinely new task.
Sam: That’s a useful test to carry back to the lab.