Interpretability demonstrations often show that intervening on a group of neurons changes a model's answer. A recent example removes an 18-neuron group in Qwen3-32B and the model answers "Simba" instead of "Scar" to a two-hop Lion King riddle. Such a demonstration establishes that the group matters, but not how much of it is necessary, whether other groups could do the same, or whether the effect carries over to other inputs. We ask: when intervening on a group of neurons changes a model's answer, how much of that group is actually necessary? For every selected group we measure all 4,096 subsets of 12 MLP neurons. On a two-alternative capital task (Qwen3-0.6B), zero-ablating 12 selected neurons lowers the answer margin by 27% while random neurons barely change it, yet replacing the same neurons with their values from other questions also barely changes it: their activations are nearly constant across questions, and we find no evidence that they carry question-specific content. On two-hop relation chains, patching 12 neurons with activations from the first-hop question shifts the answer to the bridge entity (from Scar to Simba): the greedy answer starts with the bridge in 9/11 chains for Qwen3-1.7B, 9/11 for Llama-3.2-1B and 8/10 for Gemma-3-1B; random neurons never do. The full group is rarely necessary: a few neurons often suffice, and in many chains disjoint subsets each flip the answer on their own, in an exploratory analysis most often when the model is less confident. Group effects transfer to rephrasings of the same chain but are much weaker on other tested chains. We study neuron-level causal interventions and alternative subsets; we do not recover the full connectivity of circuits.
Alex: Making a model change its answer is not the same as explaining how it answered. A new experimental study finds that selected neuron groups can contain several smaller groups that produce the same change.
Sam: So an intervention can work without identifying a unique explanation. Whose study is this, and what does it test?
Alex: It’s experimental work by Ömer Tekin Yavuz, titled “Are Causal Neuron Circuits Unique and Transferable?” He tests selected groups inside language models, asking how much of each group actually matters.
Sam: That matters if researchers are using interventions to support claims about what a model knows. What should we listen for?
Alex: Whether changing an answer reveals specific information, or just disrupts something the model depends on. We’ll follow the control that separates those possibilities, then ask whether smaller, different groups can do the same job.
Sam: What earlier demonstration makes that distinction concrete?
Alex: The paper starts with a Goodfire example involving a Lion King riddle. A model answers “Scar.” Removing a selected neuron group changes its answer to “Simba,” while removing random neurons leaves it largely unaffected.
Sam: That sounds like genuine evidence that the selected group matters. Where does the interpretation outrun the experiment?
Alex: It is genuine causal evidence, and Yavuz explicitly acknowledges that. But the whole-group result doesn’t establish whether every member is necessary. Nor does it show whether different members could independently produce the same change.
Sam: Had researchers already shown that explanations of neural networks might not be unique?
Alex: Yes. The paper discusses exhaustive work on small toy networks, and structurally different circuits found in language models. Its contribution is to examine every subset within the particular neuron group that an intervention demonstration might report.
Sam: How does that become manageable in a pretrained language model?
Alex: By limiting each selected group to twelve neurons. That gives 4,096 possible subsets, all of which the study tests. It’s exhaustive within the group, not across the model.
Sam: Before we get to the results, what counts as a neuron here? And where does the intervention happen?
Alex: A neuron here is a coordinate in an internal feed-forward layer’s activity. The intervention changes that activity only at the last prompt position, where the model begins generating its answer. Earlier positions remain untouched.
Sam: So the experiment has a tightly defined target. What does the first study reveal?
Alex: It uses capital-city questions with lettered answer options in a small Qwen model. The selected neurons are chosen for their response to zero ablation, meaning setting their activity to zero. The outcome is the margin: the model’s score advantage for the correct option over the alternative.
Sam: If those neurons really matter, how much does removing them change that advantage?
Alex: Zeroing the selected group reduces the margin by twenty-seven percent, compared with almost no change for random neurons. The effects also add up almost perfectly, and carry over to reworded questions and new capital questions.
Sam: That sounds like a successful result by several usual checks. What makes the interpretation less straightforward?
Alex: Replace the same neurons’ activity with their activity from other capital questions, and almost nothing happens. Their activity is nearly constant across those questions. Removing it disrupts a signal the answer depends on, but swapping it provides no evidence of question-specific content.
Sam: So the model can depend on that signal without the signal distinguishing Paris from another capital. That’s a different claim from identifying neurons carrying the relevant fact.
Alex: That’s the distinction supported here. Random neurons ask whether this group is unusually influential. Replacing activity from other questions asks whether its particular values distinguish the tested inputs. The group passes the first test, but offers no evidence on the second.
Sam: How does the second study make the alternative answer more informative?
Alex: It uses questions that require linking two facts. In the Lion King example, the chain goes from The Lion King to Simba, then from Simba to Scar. Simba is the bridge entity: the intermediate answer connecting the facts.
Sam: If a model doesn’t know those facts already, changing its answer would be hard to interpret. How did the author handle that?
Alex: Each included chain had to pass checks on both individual facts and the composed question. Across the initial questions, the models generally knew the individual facts but solved only about half the composed questions. The analysis therefore focuses on chains they could already answer correctly.
Sam: What intervention turns a correct composed answer into that intermediate answer?
Alex: Activation patching: copying internal activity from one prompt into another. Here, the source is the first-hop question, whose answer is Simba. The selected activity is copied into the composed question, whose correct answer is Scar.
Sam: Does that produce Simba in actual generated text, or only change the relative scores?
Alex: Both are checked. In the Qwen model, generated answers begin with the bridge in nine of eleven included chains. Random neurons patched the same way never flip an answer.
Sam: Is that confined to Qwen, or does the pattern appear elsewhere?
Alex: The broad pattern also appears in Llama and Gemma. But there’s a qualification: activity from an unrelated chain can also damage the correct answer. So the study does not establish that the selected group exclusively represents the matching bridge.
Sam: That separates steering toward Simba from merely weakening Scar. What happens when you stop treating the selected group as an indivisible unit?
Alex: For the Lion King chain in Qwen, four patched neurons suffice to flip the answer. Across chains, a handful often does the job. Effects generally accumulate smoothly rather than appearing only when the full group is patched.
Sam: Smaller subsets could all depend on one indispensable neuron. Does the study find genuinely separate alternatives?
Alex: In many chains, yes: non-overlapping subsets each cross the answer-flip threshold. But those disjoint alternatives were identified using answer scores, not checked by generating text. And they’re alternatives within the selected group, not a search through every possible mechanism.
Sam: A model already close to changing its answer might admit more alternatives. Does confidence affect the result?
Alex: The exploratory analysis suggests it does. Disjoint alternatives are more common when the original answer has a smaller score advantage. But the confidence cut-off was chosen after inspecting results, and confidence is entangled with which chain was tested.
Sam: What about transfer? Do these neurons keep doing the same thing when the wording changes?
Alex: Subset effects remain strongly aligned across rephrasings of the same chain. But applying the same selected neurons to other chains has much weaker effects, even using each target chain’s own first-hop activity. These are not general-purpose reasoning switches.
Sam: What’s the biggest reason not to generalize this into a claim about how language models reason?
Alex: The search is exhaustive only inside the selected group. Selection favors neurons with individual effects and may miss neurons that matter only together. The experiments also use small models and few chains, and they don’t recover the connections of a complete circuit.
Sam: Given those limits, who should read the full document, and where should they start?
Alex: Anyone using neuron interventions to make claims about model mechanisms. Start with Study One’s comparison between zeroing activity and replacing it from other questions. Then read Study Two’s analyses of sufficient subsets and transfer, alongside the limitations.
Sam: And for researchers who only need the broader lesson?
Alex: Changing an answer shows influence, not a complete explanation. Ask what smaller group would suffice, what other group could substitute, and whether the activity carries the information you claim.
Sam: Those are useful questions to carry into your next paper.