HINDCAST: Replaying Prediction Markets to Evaluate LLM Forecasters

The Problem with Current Forecasting Benchmarks

Standard methods for evaluating language models on forecasting tasks are prone to data leakage. Because models are often trained on data that includes the outcomes of past events, they can simply 'recall' the correct answer rather than demonstrating genuine foresight. Furthermore, retrieval-augmented models can access post-event reports, turning a forecasting task into a simple lookup. This creates a false impression of a model's ability to reason under uncertainty, as the evaluation measures memory rather than predictive capability.

The Hindcast Approach

To address these leaks, the authors introduce Hindcast, an evaluation protocol that forces models to act as if they are operating at a specific past time ($t_0$), before an event's outcome was known. The protocol uses two key components:

1. A Frozen Archive: A temporally pinned, immutable snapshot of public Reddit data. This ensures that the model can only access information that was available at the chosen $t_0$, preventing it from seeing post-event commentary.
2. Matched-Time Benchmarking: The model's forecasts are scored against both the actual resolved outcome and the market-implied probability from a prediction market (e.g., Polymarket) at the same $t_0$. This provides a human-derived baseline for what was knowable at that time.

Findings on Retrieval-Grounded Forecasting

By testing nine open-weight LLMs, the authors found that retrieval generally improves Brier scores—a measure of probabilistic accuracy—for most models. However, the effectiveness of retrieval is highly dependent on the information environment. When the Reddit archive contains substantive, pre-event discussion, retrieval significantly boosts performance. Conversely, when the archive contains only speculative or noisy chatter, retrieval often backfires, leading to worse forecasts than a zero-shot baseline that relies solely on the model's internal parametric memory.