Tao Feng, Fangxu Yu, Haozhen Zhang, Zhongjie Dai, Liangqi Yuan, Zijie Lei, Weizhi Zhang, Kunlun Zhu, Haodong Yue, Keyang Xuan, Ge Liu, Jiaxuan You
9 min
Abstract
No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment. Existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult. We present a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals, covering single-turn, multi-turn, and personalized routing. Based on this formulation, we develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks. We further introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Our empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.
Alex: How do they actually test whether this unified approach works?
Sam: They built an automated evaluation pipeline and a benchmark called xRouteBench. It covers a wide range of task types — general knowledge questions, long back-and-forth conversations, visual data like charts, time-series numerical data, and even personalized scenarios where the right answer depends on who's asking.
Alex: So it's not just testing on simple trivia. It includes the messy, real-world stuff.
Sam: Exactly. And using that benchmark alongside their open-source library, they found that learned routing — where the system improves through experience — produces meaningfully better results than just always picking the same model, while keeping costs under control.
Alex: What happens when the question isn't text at all — like when someone uploads a chart or an image?
Sam: The system handles that by converting non-text inputs into plain text descriptions before the routing decision is made. Charts get rendered into numerical summaries. Images get described in words. So every candidate model receives a standard text query, regardless of what the original file looked like.
Alex: That's clever — it keeps the playing field level across all the models.
Sam: Precisely. It means differences in performance reflect actual reasoning ability, not just which model happened to be better at reading a particular file format. For video, the system samples frames across aligned time windows and uses a vision-language model to describe the motion and objects into a merged text query.
Alex: What about personalisation — situations where the right answer genuinely depends on who's asking?
Sam: The framework supports that too. It incorporates user profiles and interaction histories so the routing policy can predict which response a specific person would prefer. And to generate training data for those preferences without asking humans to grade thousands of responses, they use an external language model — conditioned on simulated user personas — to judge pairs of candidate answers and record which one wins.
Alex: So the system learns individual taste without needing a huge team of human raters.
Sam: That's the idea. And once all that supervision data is gathered, the routers themselves come in several distinct designs. The simplest baselines just pick the smallest or largest model by parameter count, without looking at the question at all — useful as a reference point, but not very smart.
Alex: And the more sophisticated ones?
Sam: Single-turn routers look at the question once and make a selection using techniques like nearest-neighbour matching or learned classifiers. Multi-turn routers go further — they treat the conversation history and intermediate responses as part of the decision, allowing the system to iteratively consult specialists or break a complex query into sub-problems before combining the final answer.
Alex: That multi-turn approach sounds like it could get complicated fast. How do researchers actually build and test these networks without losing track of everything?
Sam: The library includes a visual canvas — essentially a flowchart interface where every processing step appears as a connected block. You drag lines between boxes to wire data sources to routers to evaluation outputs. Two input blocks supply the dataset and the model pool, feeding into a data engine that produces a performance-cost grid, and each router block plugs into that grid.
Alex: And if you want to swap out one routing algorithm for another?
Sam: You replace a single box on the screen. Every parameter is automatically mirrored from the underlying configuration files, so the visual interface and the command-line version run identical experiments. And if you run the same experiment twice, the system recognises that nothing has changed and skips the expensive response-collection step entirely.
Alex: That saves a significant amount of compute during testing.
Sam: It does. And that efficiency matters, because comparing sixteen different router designs across multiple benchmarks would otherwise be prohibitively slow.
Alex: The paper also tests how these routers fit into larger multi-agent setups, right? Where multiple AI systems collaborate on a single task?
Sam: Yes. They test five different coordination layouts — think of an office team where a manager splits a big project among workers and then combines their reports. They use star, tree, graph, and chain structures, plus a layout where a planner generates three sub-queries and a summariser merges the final answers. In every case, each node in the network is replaced by a router that selects the right model for that specific task call.
Alex: So the routing logic scales all the way up from a single question to a full team of collaborating agents.
Sam: Exactly. And for live deployment, the system connects to real communication platforms. In one tested setup, a user sends a question through Slack, the server selects two models from a pool to generate answers side by side, and the user clicks to indicate which response they preferred — or marks them as equal. Every turn in a multi-turn conversation gets labelled the same way, building up a record of real human preferences over time.
Alex: What are the main limitations to keep in mind?
Sam: Multi-turn routing and agentic delegation are both sensitive to the quality of the base model doing the planning. If that model isn't capable enough to decompose a complex problem correctly, the system introduces overhead that can outweigh any cost savings. That dependence on the planning model is a clear boundary for how far automation can reach right now.
Alex: So the router is only as good as the model helping it plan.
Sam: That's a fair summary. The paper is candid about it. And looking ahead, the authors suggest future systems will likely adapt their candidate pools and cost-performance thresholds in real time — responding to live changes in API latency and shifting user preferences — rather than relying on fixed rules set at deployment.
Alex: Dynamic adaptation instead of static configuration. That does seem like the natural next step.
Sam: The unified infrastructure the library provides gives researchers a solid, fair foundation for comparing these methods and building toward that future. The work is careful to frame these as promising directions rather than solved problems — which is the honest position given where the field currently stands.
Alex: A useful piece of infrastructure for a genuinely difficult problem. Thanks for walking us through it, Sam.
Sam: Thanks for listening to ResearchPod.