Most of the LLM routing gap is task type

An LLM router aims to improve accuracy and efficiency by dynamically selecting the best language model for each specific query. However, recent studies show that existing routers fall far short of the theoretical performance ceiling and often fail to outperform the single best model overall. This paper investigates what missed queries have in common by constructing and executing a comprehensive test matrix. The author evaluates 14 distinct models across 294 total queries, covering seven task types in three different languages (Korean, English, and Hindi). Every model answers every query twice under identical deterministic configurations to measure both performance and execution stability.

By evaluating the matrix, the paper analyzes the gap between the optimal oracle router and the best single model. When counting an answer as correct only if a model succeeds in both execution runs, the oracle-to-best gap yields 29 actionable items. Assigning each task type to a single, pre-selected model in advance resolves 21 of those 29 items. Adding a language split resolves two additional items, leaving only six items unoptimized. This demonstrates that coarse-grained task type accounts for the overwhelming majority of the addressable routing gap.

Running the identical matrix twice reveals a surprising degree of non-determinism, with 5.37% of model-query cells flipping their correct/incorrect status between runs despite a temperature of zero and a fixed seed. Total billed costs similarly fluctuate by nearly 4% between identical runs. Because this run-to-run volatility is comparable in magnitude to the residual routing gains, the author cautions that small performance margins in routing studies may simply reflect execution noise rather than genuine algorithmic superiority.