Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure. They do not account for an opponent that might be unknown, and that can adapt over time and shift the feedback signal. To address this, we extend the game-theoretic evaluations and define strategic selection as the meta game layered on top of the base game. In each round, a strategic selector fields an LLM from a portfolio of models to navigate a game payoff matrix that is latent and learned online. We find that the evaluation signal is game-specific, and that extracting a reliable signal requires a much higher number of seed runs $N$ compared to traditional benchmarking setups. Evaluating across 5 games and 10 LLMs, we show a model's strategic competence varies significantly depending on the game setup and the opponent faced. Building on this, we implement a suite of selector methods and find that approaches that rely on query embeddings or LLM self-reflection do not consistently capture the underlying strategic signal. A lightweight multi-armed bandit algorithm performs more reliably across games, though it still falls short of the best fixed model. Finally, we show that simple non-stationary and adversarial bandit variants fail to outperform a stationary bandit baseline, pointing to a gap in how current methods handle a shifting opponent in strategic settings.
Sam: The strongest language model isn't necessarily the best opponent. In this study, models' strategic strengths depended on the game and whom they faced, while selection methods struggled to exploit those differences.
Alex: That changes what “best model” means. Are we talking about models answering questions, or models making decisions against each other?
Sam: Decisions against each other. This is a methods-and-experiments preprint by Arastun Mammadli and colleagues at the University of Exeter and IBM Research. Its title is “Choosing LLMs for Strategic Agent Interactions.”
Alex: So if I'm building agents that negotiate or compete, a general capability ranking might not tell me which model to deploy.
Sam: That's the motivation. The study suggests that model choice also selects a style of strategic behavior. We'll unpack how the authors tested that, why a simple selection method beat more elaborate alternatives, and where the evidence remains thin.
Alex: Researchers already test language models in games, and software already chooses between models. What was missing between those approaches?
Sam: The authors identify a gap between them. Strategic benchmarks generally keep each player's model fixed. Conventional model routers, which assign requests to models, usually balance answer quality against cost without accounting for an opponent.
Alex: An opponent can change in response to you. So learning which model worked yesterday doesn't necessarily identify which model will work tomorrow.
Sam: That makes the problem non-stationary: the conditions you're learning about can change over time. The authors put a strategic selector above the models. It's a policy that chooses which model plays each round.
Alex: What does that selector actually see? If it knows the opponent's model, this could become a lookup problem.
Sam: In the main selection setup, it doesn't know the opponent's model or selection policy. It sees its own chosen model, that model's action, and the resulting gain. It also doesn't see what would have happened had it chosen another model.
So there are two decisions: choosing the player, then letting that player choose its move.
Sam: The authors call the upper layer a meta game: a game of choosing models that play another game. Consider Cournot Duopoly, where two firms choose production quantities. Their combined output determines market price and their profits.
Alex: In that example, the selector chooses a model, not a production quantity. It learns from the profit that model's quantity helped produce.
Sam: With a cost adjustment. The study's main selection score combines normalized game utility with normalized model usage cost. Those components receive equal weight, so the highest-profit model isn't automatically the highest-scoring choice.
Alex: Before testing selectors, they need evidence that the models offer meaningfully different choices. How broad was that comparison?
Sam: They tested ten open-weight models, drawn from several model families and sizes. Each faced the others across five games. The inventory spans resource allocation, quantity competition, coordination, auctions, and brinkmanship.
Alex: Did those games produce one consistent ranking, or genuinely different strengths?
Sam: Different strengths. Llama models led in quantity competition but weren't among the leaders in Chicken, the brinkmanship game. Overall, model size had only a weak positive association with game utility; once usage costs entered the score, the association was moderately negative.
Alex: Size could hide differences in behavior. Did they examine what the models actually did, rather than just their final scores?
Sam: They measured game-specific dispositions, meaning patterns in the models' actions. Even models from the same family differed sharply. In Chicken, the middle-sized Granite model almost always chose to dare, while its smaller and larger relatives never did.
Alex: That makes the selection problem concrete. You're not just choosing more or less reasoning capacity; you're choosing a tendency to escalate or concede.
Sam: Within these tested games, yes. Another example is the second-price auction, where the winner pays the next-highest bid. The smallest Granite model outperformed much larger models by consistently bidding its true valuation.
Alex: Those differences could still be noisy. A single unusual answer can change a game's outcome. How did the authors separate behavior from sampling variation?
Sam: For each fixed-model matchup, they ran three hundred independent, single-round repetitions. They also examined how reliably smaller repetition budgets recovered model comparisons. The required replication depended heavily on the game.
Alex: So using the same small evaluation budget everywhere could give rankings with very different reliability.
Sam: That's their warning. Quantity competition separated models most clearly. The resource-allocation game, Colonel Blotto, separated them least clearly because the models played similarly. More repetitions can't create a large performance gap where little exists.
Alex: Let's move to the selection result. Once those behavioral differences existed, which method actually learned to use them?
Sam: Against fixed but unknown opponents, Thompson sampling had the highest cost-adjusted gain among the tested selectors in every game. It's a lightweight learning rule. It maintains an uncertain reward estimate for each model, samples from those estimates, and chooses the model with the highest draw.
Alex: What did that beat? A simple rule winning only matters if the alternatives were credible.
Sam: The alternatives included routers that encode context into vectors for similarity matching, and a language-model selector using reflection over past outcomes. The reflection-based selector performed worst overall. The authors interpret this as difficulty extracting the strategic signal from context, rather than evidence that more elaborate methods necessarily help.
Alex: Best selector isn't the same as best deployment choice. Did it beat simply sticking with a well-chosen model?
Sam: It still fell short of the best fixed model. Even in quantity competition, where its advantage over other selectors was strongest, a fixed Llama model did better. So the study demonstrates selection headroom, not a solved selection problem.
Alex: And the harder case is an opponent that also changes models. Did the simple method survive that comparison?
Sam: In the reported adaptive-opponent comparison, which used quantity competition, it remained strongest. Its cost-adjusted gain was about seventy-two on the normalized hundred-point scale. The nearest alternative, a version that retains only recent observations, scored about sixty-six.
Alex: Forgetting old evidence sounds sensible against a changing opponent. But here, methods designed for change didn't outperform the ordinary sampler.
Sam: That's what these runs show. Discounting old observations also failed to improve on it, as did an adversarial bandit method. That doesn't establish that adaptation is useless; it shows these particular adaptations didn't capture the shifting strategic signal well enough.
Alex: You emphasized independent repetitions earlier. Were these learning methods tested with the same replication?
Sam: No, and that's the limitation I'd prioritize. Selectors played three hundred rounds per matchup, but only one independent seed per matchup to control inference costs. Many sequential rounds aren't independent reruns of the learning process, and the authors say their fixed-model noise analysis doesn't directly transfer.
Alex: So how should a researcher use this document, and where should they start?
Sam: Read it if you develop model routers, interacting agents, or strategic evaluations. Start with “Strategic LLM Selection,” then compare the fixed-opponent and adaptive-opponent results. Read the appendix on noise before treating any ranking as stable. Also remember that these models exchanged parsed game actions, not free-form negotiation, so persuasion and deployment risks remain outside the tests.
Alex: And for someone who won't read the full study, what's the line to keep?
Sam: Choosing a model means choosing strategic behavior—not just capability and price.
Alex: Thanks for listening.