It has been demonstrated that, in certain settings, large language models can perform weight-space programming to directly modify the weights of neural networks. However, it is unknown how dedicated training of these networks can contribute to the successes of weight-space programming. We present a benchmark of LLM-driven weight-space programming on vision models, in which five LLMs edit convolutional networks and vision transformers on relabeling and error-correction tasks on Waterbirds, CIFAR-100, PACS and Entity-30. We report more than 2,600 such edit processes, whereby each edit is scored on held-out images for success, side effects and accuracy loss. We compare standard training with label smoothing, gradient-based editable training, and our Edit-Ready Training (ERT), a novel method that rehearses the direct, closed-form weight edits LLMs perform. We show that preparing networks for editing raises the success of LLM edits substantially where standard networks are hard to edit. ERT further achieves the highest mean score for GPT-6 Astra and Claude Opus 5.5 across all evaluated baselines. We will release our LLM testing framework alongside all tasks, editing trajectories and scores. Project page: https://viscom.uni-ulm.de/wsp
Sam: An image classifier can be trained to be easier for a language model to change. This study found the biggest gains where normally trained classifiers resisted targeted edits.
Alex: If I want to repair one recurring mistake, that sounds useful. But does easier to change also mean fewer mistakes elsewhere?
Sam: Not necessarily, and that tension runs through the results. This is a benchmark and method preprint by Timo Ropinski and Sebastian Kranz at Ulm University. It’s called “Training Vision Models for LLM-Driven Weight-Space Programming.”
Alex: So the audience isn’t just people building language models. It’s also researchers asking how to maintain an image classifier after deployment.
Sam: Yes. The authors ask whether training should prepare a classifier for future repairs, rather than accuracy alone. We’ll unpack the repair process, what the training changes, and why better editing scores still don’t guarantee preserved accuracy.
Alex: Start with the repair itself. What does weight-space programming mean here, and how is it different from ordinary retraining?
Sam: Weight-space programming means a language model directly changes another network’s existing weights. The agent inspects internal responses, writes changes, and tests their effects. During editing, it cannot use gradients, the derivatives normally used to train networks.
Alex: So it cannot quietly turn the task into fine-tuning. What information does it get instead?
Sam: A request, a few support images with their desired labels, and reference images whose predictions should stay unchanged. But the agent never sees the pictures themselves. It sees only how the network responds to them.
Alex: Give me a request where that distinction matters. Are these arbitrary label changes, or repairs to existing errors?
Sam: The benchmark includes both. One repair asks the agent to correct kitchen utensils wrongly classified as tableware. Other tasks deliberately move a fine-grained concept into another class, or correct misclassified sketches of a particular object class.
Alex: Those require different kinds of success. How do the authors stop an agent from fixing the examples it receives while breaking everything else?
Sam: They score the submitted network on held-out images the agent never receives. Their primary measure is a net score: targeted success minus side effects. For error correction, side effects mean previously correct predictions outside the target group becoming wrong.
Alex: And if an agent can keep trying indefinitely, it might still overfit those support examples. What constrains the search?
Sam: The WeightLab sandbox limits forward tests, weight reads, and changes. Each process allows eight edit calls. Agents can undo changes, but that does not replenish their budget, and they are blinded to how the network was trained.
Alex: Now to the new training method. Existing model-editing research already tries to make repairs targeted. What was missing?
Sam: Most editing methods compute a repair with a prescribed algorithm. An earlier approach, Editable Neural Networks, trains a network so a few gradient steps can correct it without harming other predictions. Here, the authors instead rehearse the direct changes language-model agents actually tend to write.
Alex: That makes the editor’s habits central. What did they observe that was simple enough to rehearse during training?
Sam: Their preliminary study found that the agent mostly changed the classifier head, the final layer turning image features into class scores. It concentrated on features that distinguished the support images from the reference images. The larger analysis largely confirmed that pattern.
Alex: For the utensil repair, that means finding internal features that respond differently to the utensils than to the examples we want left alone.
Sam: That’s the contrast the method uses. Edit-Ready Training temporarily changes the final layer along that contrast. It strengthens the target class and weakens the source class, using either a small selection of features or a direction through feature space.
Alex: But writing that temporary repair only demonstrates that a particular change works. How does it teach the network to become more editable?
Sam: During training, separate query images check whether the repair generalizes beyond its support examples. Reference images check whether other predictions remain stable. Those checks contribute to the training loss, alongside ordinary classification accuracy.
Alex: So training rewards representations that make a later repair selective. Does the temporary edit become part of the trained network?
Sam: No. The authors backpropagate through the simulated repair into the original network, then discard the temporary change. The goal is to prepare the features, not retain that particular repair. Concepts edited in the main evaluation were excluded from the rehearsal episodes.
Alex: That’s a plausible mechanism. How broad was the test, and what comparison carries the result?
Sam: The study reports more than twenty-six hundred scored edit processes, including pilots and ablations. The main comparison spans nine combinations of dataset, task, and network architecture. It compares ordinary training, label smoothing, gradient-based editable training, and the new rehearsal method.
Alex: Label smoothing softens training targets rather than rehearsing repairs. Did the dedicated rehearsal beat that simpler change?
Sam: On average, yes, for the leading editors, GPT-6 Astra and Claude Opus 5.5. With Opus, the mean net score rose from about sixty points under ordinary training to seventy-two with Edit-Ready Training, on a hundred-point scale.
Alex: That is success minus side effects, not overall classification accuracy. Was the new method clearly better than the earlier editable-training approach too?
Sam: It had the highest mean score for both leading editors, but its average advantage over gradient-based editable training was not statistically significant. No training method won everywhere. The strongest claim is improvement over ordinary training, particularly in settings that were hard to edit.
Alex: Which also suggests a practical screening step. If ordinary training already permits a clean repair, specialized training may add little.
Sam: That matches the authors’ results. Several settings were already easy to edit, and preparation offered little average benefit there. Waterbirds and the Entity-30 confusion repair were harder, and preparation helped substantially.
Alex: Could the training simply be teaching the language model a different repair strategy?
Sam: The observed edit shapes stayed broadly similar across training methods. On Waterbirds, Opus could make smaller changes to prepared networks. That supports the authors’ interpretation that training makes an ordinary head edit sufficient, rather than inducing a new editing strategy.
Alex: Let’s return to the risk we opened with. When the net score improved, did overall test accuracy survive?
Sam: Not intact. Across training methods, the leading editors finished roughly three to four percentage points below the unedited networks on full-test accuracy. Later edits repaired some damage from the first change, but not all of it.
Alex: Deliberate relabeling complicates that measure, though. A successful requested change can disagree with the original test label.
Sam: The authors explicitly note that. Relabeled concepts count as errors against the original labels, although they are a small part of the test set. Accuracy also fell on the error-correction tasks, so better net scores should not be read as damage-free maintenance.
Alex: How much confidence should we place in the result beyond these particular networks and editors?
Sam: Most training-method comparisons use a single trained network per setting, so the intervals mainly describe variation between editing processes. A seed check found little network-to-network variation in the tested subset. Still, the study covers image classification only, and weaker editors did not reliably benefit from preparation.
Alex: So who should read the full document, and where should they start?
Sam: Researchers building model-editing agents or training maintainable vision models should read it. Start with Main Results to see where preparation helps and where it doesn’t. Then read Edit-Ready Training for the mechanism, and the accuracy breakdown in Additional Results before choosing a method.
Alex: And what should everyone else carry away?
Sam: How you train a model can shape how well you can repair it later. But easier to edit does not yet mean safe to edit.
Alex: Keep both halves of that result in mind.