Arastun Mammadli, Herbert Woisetschläger, Jawad Fayaz, Shiqiang Wang
8 min
Abstract
Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capability dimensions. However, they only evaluate strategic performance across individual fixed models. In parallel, contemporary LLM routers pick models under quality-cost trade-offs, but with no notion of a strategic structure. They do not account for an opponent that might be unknown, and that can adapt over time and shift the feedback signal. To address this, we extend the game-theoretic evaluations and define strategic selection as the meta game layered on top of the base game. In each round, a strategic selector fields an LLM from a portfolio of models to navigate a game payoff matrix that is latent and learned online. We find that the evaluation signal is game-specific, and that extracting a reliable signal requires a much higher number of seed runs $N$ compared to traditional benchmarking setups. Evaluating across 5 games and 10 LLMs, we show a model's strategic competence varies significantly depending on the game setup and the opponent faced. Building on this, we implement a suite of selector methods and find that approaches that rely on query embeddings or LLM self-reflection do not consistently capture the underlying strategic signal. A lightweight multi-armed bandit algorithm performs more reliably across games, though it still falls short of the best fixed model. Finally, we show that simple non-stationary and adversarial bandit variants fail to outperform a stationary bandit baseline, pointing to a gap in how current methods handle a shifting opponent in strategic settings.
Sam: With a cost adjustment. The study's main selection score combines normalized game utility with normalized model usage cost. Those components receive equal weight, so the highest-profit model isn't automatically the highest-scoring choice.
Alex: Before testing selectors, they need evidence that the models offer meaningfully different choices. How broad was that comparison?
Sam: They tested ten open-weight models, drawn from several model families and sizes. Each faced the others across five games. The inventory spans resource allocation, quantity competition, coordination, auctions, and brinkmanship.
Alex: Did those games produce one consistent ranking, or genuinely different strengths?
Sam: Different strengths. Llama models led in quantity competition but weren't among the leaders in Chicken, the brinkmanship game. Overall, model size had only a weak positive association with game utility; once usage costs entered the score, the association was moderately negative.
Alex: Size could hide differences in behavior. Did they examine what the models actually did, rather than just their final scores?
Sam: They measured game-specific dispositions, meaning patterns in the models' actions. Even models from the same family differed sharply. In Chicken, the middle-sized Granite model almost always chose to dare, while its smaller and larger relatives never did.
Alex: That makes the selection problem concrete. You're not just choosing more or less reasoning capacity; you're choosing a tendency to escalate or concede.
Sam: Within these tested games, yes. Another example is the second-price auction, where the winner pays the next-highest bid. The smallest Granite model outperformed much larger models by consistently bidding its true valuation.
Alex: Those differences could still be noisy. A single unusual answer can change a game's outcome. How did the authors separate behavior from sampling variation?
Sam: For each fixed-model matchup, they ran three hundred independent, single-round repetitions. They also examined how reliably smaller repetition budgets recovered model comparisons. The required replication depended heavily on the game.
Alex: So using the same small evaluation budget everywhere could give rankings with very different reliability.
Sam: That's their warning. Quantity competition separated models most clearly. The resource-allocation game, Colonel Blotto, separated them least clearly because the models played similarly. More repetitions can't create a large performance gap where little exists.
Alex: Let's move to the selection result. Once those behavioral differences existed, which method actually learned to use them?
Sam: Against fixed but unknown opponents, Thompson sampling had the highest cost-adjusted gain among the tested selectors in every game. It's a lightweight learning rule. It maintains an uncertain reward estimate for each model, samples from those estimates, and chooses the model with the highest draw.
Alex: What did that beat? A simple rule winning only matters if the alternatives were credible.
Sam: The alternatives included routers that encode context into vectors for similarity matching, and a language-model selector using reflection over past outcomes. The reflection-based selector performed worst overall. The authors interpret this as difficulty extracting the strategic signal from context, rather than evidence that more elaborate methods necessarily help.
Alex: Best selector isn't the same as best deployment choice. Did it beat simply sticking with a well-chosen model?
Sam: It still fell short of the best fixed model. Even in quantity competition, where its advantage over other selectors was strongest, a fixed Llama model did better. So the study demonstrates selection headroom, not a solved selection problem.
Alex: And the harder case is an opponent that also changes models. Did the simple method survive that comparison?
Sam: In the reported adaptive-opponent comparison, which used quantity competition, it remained strongest. Its cost-adjusted gain was about seventy-two on the normalized hundred-point scale. The nearest alternative, a version that retains only recent observations, scored about sixty-six.
Alex: Forgetting old evidence sounds sensible against a changing opponent. But here, methods designed for change didn't outperform the ordinary sampler.
Sam: That's what these runs show. Discounting old observations also failed to improve on it, as did an adversarial bandit method. That doesn't establish that adaptation is useless; it shows these particular adaptations didn't capture the shifting strategic signal well enough.
Alex: You emphasized independent repetitions earlier. Were these learning methods tested with the same replication?
Sam: No, and that's the limitation I'd prioritize. Selectors played three hundred rounds per matchup, but only one independent seed per matchup to control inference costs. Many sequential rounds aren't independent reruns of the learning process, and the authors say their fixed-model noise analysis doesn't directly transfer.
Alex: So how should a researcher use this document, and where should they start?
Sam: Read it if you develop model routers, interacting agents, or strategic evaluations. Start with “Strategic LLM Selection,” then compare the fixed-opponent and adaptive-opponent results. Read the appendix on noise before treating any ranking as stable. Also remember that these models exchanged parsed game actions, not free-form negotiation, so persuasion and deployment risks remain outside the tests.
Alex: And for someone who won't read the full study, what's the line to keep?
Sam: Choosing a model means choosing strategic behavior—not just capability and price.
Alex: Thanks for listening.