Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning

Traditional mechanistic interpretability is retrospective, meaning it analyzes models that have already learned a task. This creates a chicken-and-egg problem for parameter-efficient fine-tuning (SFT): researchers want to identify and tune only the task-critical parameters, but those parameters only become critical after the model has learned the task. This paper asks: can we predict the task-critical mechanisms of a future, post-SFT model using only the current, pre-SFT model parameters?

The authors introduce a forward-looking localization framework that treats SFT as a continuous parameter evolution. They decompose the transition from the initial model to the ideal post-SFT model into two steps: isolating the initial gradient direction and extrapolating the distance of the parameter shift.

1. Directional Estimation: They model an infinitesimal update toward the target task using a first-order Taylor expansion, allowing them to capture the gradient direction without performing a full fine-tuning run.
2. Distance Extrapolation: They conceptualize the total parameter shift as an accumulation of infinite recursive updates. By leveraging a step scalar (K) and a translation operator, they estimate the sensitivity of the model's internal components to the final, ideal state.

Practically, the method requires only a "probing" SFT step—fine-tuning on just 1% of the target data for a single epoch—to derive the necessary gradient information. The authors implement this at both the neuron and component levels, the latter of which integrates with LoRA to dynamically allocate ranks based on predicted importance.

Experiments on Mistral-7B, LLaMA-2-13B, and Qwen3-30B show that this predictive approach consistently outperforms existing localization baselines in both Target Task Accuracy (TTA) and the retention of general capabilities (PTA). The method is particularly robust for complex tasks where the representational gap between the pre- and post-SFT model is large. Furthermore, the authors demonstrate that their approach scales efficiently with model size and effectively mitigates inter-task conflicts in multi-task settings by proactively avoiding the allocation of shared, polysemantic neurons.