FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches

Research Approach

This study introduces WC2026-Agents, a contamination-free benchmark designed to evaluate Large Language Models (LLMs) as autonomous forecasting agents. By utilizing the 104 matches of the 2026 FIFA World Cup—all occurring after the models' training cutoffs—the researchers created a real-world environment where the ground truth is unambiguous and future-dated. Four frontier models (Claude Opus 4.8, ChatGPT, Gemini 3.1 Pro, and Grok) were tasked with a standardized search-act-reflect loop: gathering web evidence, committing to a 1X2 probability distribution, placing a virtual $100 bet, and reflecting on their performance after the match outcome.

Key Findings

The researchers found that frontier models are remarkably interchangeable when it comes to raw prediction. The agents issued identical top picks in 92% of matches and none managed to beat the betting market's Brier score. However, these models diverge sharply when evaluated as decision-makers. Betting return-on-investment (ROI) varied significantly, ranging from -18% to +10%, driven by differences in staking discipline and the tendency to 'fade' (bet against) the market. While some models consistently cited market odds in their reasoning, others relied almost exclusively on football-specific metrics, yet both approaches converged on similar predictions. Furthermore, the agents showed distinct 'behavioral fingerprints' in their post-match reflections, with some models being far more prone to self-justification or softening their errors than others.

This benchmark demonstrates that accuracy is an insufficient metric for evaluating agentic behavior. By pairing LLMs with an economically grounded human baseline—the betting market—the study reveals that models can possess similar predictive capabilities while maintaining vastly different internal reasoning processes and decision-making strategies. This highlights the importance of evaluating LLMs on axes like calibration, staking discipline, and self-knowledge, which are critical for real-world applications where models must act on their beliefs rather than just providing a single answer.