Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Fine-tuning large language models (LLMs) to incorporate new knowledge often results in a phenomenon the authors call the Knowing-Using Gap. While models can quickly memorize new factual triplets (e.g., "Sydney is in Australia"), they frequently fail to apply this knowledge to downstream reasoning tasks (e.g., "What is the capital of the country where Sydney is located?"). This gap is characterized by two metrics: an accuracy gap, where generalization performance lags behind memorization, and a temporal lag, where generalization emerges significantly later than memorization during the training process.

To understand why this happens, the researchers developed a technique called self-patching. This method involves copying hidden state representations of an entity from a source prompt (where the model has memorized the fact) and injecting them into a target prompt (where the model is failing to reason) at different layers. By scanning all layer pairs, the authors created a spatial map of how injected knowledge permeates the model. They found that even when a model fails to generalize, the necessary information is often already present internally but is "stranded" in layers that do not interact with the model's reasoning circuits.

Based on these findings, the authors propose the knowledge-circuit misalignment hypothesis. This suggests that fine-tuning encodes new facts into storage states that support simple recall but are not automatically integrated into the mid-layer computations required for multi-hop reasoning. The authors demonstrate that this is not a permanent failure; by manually relocating these representations to the correct computation-effective layers, they can trigger successful reasoning. They further developed a non-oracle heuristic strategy based on these findings, which successfully recovers 58-75% of the potential generalization performance that would otherwise be lost.