When Should LLMs Search? Counterfactual Supervision for Search Routing

Search-augmented language models often face a dilemma: should they rely on their internal parametric knowledge or use an external search tool? While search can help resolve factual gaps, it can also be unnecessary for questions the model already knows or counterproductive for questions requiring clarification or premise correction. The author investigates whether models can learn to perform instance-level search routing—deciding for each specific question whether to search or not—to improve overall task success.

To avoid relying on subjective human labels for search usefulness, the author constructs a counterfactual oracle. For every question, the model generates two traces: a no-search trace and a forced-search trace. By evaluating the success of both, the author assigns each instance a label: NOSEARCH (if the model succeeds without search), SEARCH (if the model fails without search but succeeds with it), or UNSOLVED (if both fail). This oracle serves as a training signal for supervised fine-tuning and preference optimization, teaching the model to align its first-action decision with the oracle-consistent path.

Training on these counterfactual outcomes significantly improves search-routing performance. For Gemma E2B, the macro-F1 score on oracle-eligible examples rose from 0.7082 to 0.8235, while Qwen3.5-4B improved from 0.7053 to 0.8365. The analysis reveals that models have different initial routing tendencies: Gemma primarily needed to learn restraint to avoid over-searching, whereas Qwen needed to reduce both over-searching and missed searches. The study also identifies that UNSOLVED cases represent heterogeneous bottlenecks—such as model capacity, retrieval limits, and evidence synthesis—that cannot be solved by routing decisions alone.

This work shifts the focus of tool-use research from simply calling tools correctly to deciding whether a tool is needed at all. By grounding the routing decision in observed task outcomes rather than heuristic prompts, the framework provides a robust, model-specific method for optimizing the efficiency and accuracy of search-augmented systems.