LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports

Evaluating the forecasting capabilities of Large Language Models (LLMs) is notoriously difficult because most benchmarks are static or retrospective, allowing models to potentially memorize outcomes or leverage information leaked from training data. To address this, the authors introduce LLM-SoccerArena, a prospective, live benchmarking platform designed to test how well LLMs synthesize information to predict unresolved real-world events.

The researchers implemented a factorial benchmark design that systematically varies four key dimensions: model version, information access (web search vs. internal knowledge), prompting strategy, and forecast horizon (e.g., 24 hours vs. 2 hours before kickoff). By focusing on soccer—a domain with objective, verifiable outcomes and continuous event streams—the platform ensures that all forecasts are recorded before the match results are known, effectively eliminating the risk of data contamination.

Through a large-scale evaluation of the 2026 FIFA World Cup, the authors analyzed 104 matches and 15 tournament-related questions across seven state-of-the-art LLMs. The results indicate that while LLMs are capable of generating structured, probabilistic forecasts, their performance is highly sensitive to the experimental conditions. A notable finding is that providing models with web access improves their Brier score (a measure of probabilistic accuracy) by only a small margin (0.023), suggesting that current search-augmented capabilities provide only modest benefits for this type of predictive task.

LLM-SoccerArena provides a robust, standardized, and auditable framework for assessing the real-world reasoning and forecasting abilities of AI models. By moving away from static datasets toward a live, prospective evaluation protocol, the platform offers a more reliable way to measure how models handle uncertainty and evolving information. This infrastructure is designed to be continuously updated, making it a valuable tool for researchers interested in the ongoing evolution of LLM performance in dynamic, real-world environments.