In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper about how large language models find information in long documents.
Sam: We're discussing a method called Logit-Contribution Scoring, or LOCOS. The central puzzle is this: while researchers thought they understood how these models "look" for answers, they'd been ignoring what the models actually write.
Alex: So this paper is asking whether we've been looking at the wrong part of the model's brain?
Sam: Exactly. The standard approach identifies important components by seeing where the model "reads." But this study argues that reading isn't the same as producing an answer.
Alex: Okay, so if I'm reading a book, I might glance at a page, but that doesn't mean I'm writing down a useful summary. Is that the distinction?
Sam: That's it. Think of it this way: inside a language model, there are many small processing units called attention heads. Each one has two jobs. The first job is deciding where to look in the text — scanning for relevant words, like your eyes moving across a page. The second job is deciding what to do with what it found — actually writing something useful into the model's internal workspace. Most existing methods only track the first job. LOCOS tracks both.
Alex: And if we only track where it looks, we miss the heads that are actually synthesizing information into an answer?
Sam: Precisely. Existing methods assume that if a model looks at a word, it must be copying that word into its answer. But for complex questions, it reasons. It might look at "Eiffel Tower" but write "France" into its internal state — because it understood the connection, rather than just copying the text.
Alex: So if we only look for the "copy-pasters," we're completely blind to the "synthesizers." How does LOCOS actually see those other heads?
Sam: LOCOS asks a direct question about each head: does this head's output actually push the model toward the correct answer? It does this by comparing what the head writes against the direction of the right answer in the model's internal space. If a head's contribution lines up with the correct answer, it gets a high score.
Alex: But what if a head just happens to write something that accidentally points toward the right answer? Doesn't that risk giving credit where it isn't due?
Sam: That's where spatial contrast comes in. The researchers don't just look at the score in isolation — they compare it against the background noise from the rest of the text. If a head only scores highly when it's focused on the relevant piece of information, and not otherwise, that's strong evidence it's doing genuine retrieval work. It's like checking whether a librarian actually handed you the right book, rather than just noticing they were standing near the right shelf.
Alex: And when they tested this — when they removed the heads LOCOS identified as important — what happened?
Sam: The model's ability to answer questions collapsed much faster than when they used older methods to select which heads to remove. That's the key validation. It shows LOCOS is finding the heads that genuinely matter, not just the ones that look busy.
Alex: Did removing those heads affect anything else, like basic arithmetic or general knowledge?
Sam: The effect was specific to retrieval. The model's ability to do arithmetic or recall facts it already knew stayed intact. That specificity is meaningful — it suggests LOCOS is isolating a distinct piece of machinery, not just disrupting the model broadly.
Alex: You mentioned a benchmark called NoLiMa. What makes that test particularly useful here?
Sam: NoLiMa is designed to test exactly the cases where copy-pasting fails. The answer to a question doesn't share any words with the relevant part of the source text — so the model genuinely has to reason rather than pattern-match. Older methods that rely on token-matching struggle badly on these tasks. LOCOS, because it tracks what the model writes rather than just where it looks, successfully identifies the heads doing that underlying reasoning.
Alex: So the older tools were essentially blind to an entire category of thinking the model was doing.
Sam: That's a fair way to put it. And the practical implications go beyond just understanding the model better. If you can precisely identify which components handle retrieval, you can start to optimize how models use memory, or improve how you diagnose failures — figuring out why a model got something wrong, rather than just observing that it did.
Alex: That shift from observation to diagnosis feels like it matters quite a bit for building more reliable systems.
Sam: It does. There's a meaningful difference between knowing a model gave a wrong answer and knowing which part of its reasoning broke down. LOCOS is a step toward that second kind of understanding — and that's what makes it worth paying attention to.
Alex: Thanks for listening to ResearchPod.