Forecasting With LLMs: Improved Generalization Through Feature Steering

When large language models (LLMs) perform forecasting, they often suffer from "look-ahead bias," where the model inadvertently uses information from its training data that would not have been available at the time the forecast was supposedly made. This paper investigates whether this bias is merely a result of memorized facts or if it is tied to specific, identifiable internal representations. The authors ask if they can isolate and manipulate these internal states to force the model to reason only with information available at a specific historical point in time.

The researchers use sparse autoencoders (SAEs) to decompose the internal activations of LLMs into interpretable features. By analyzing how these models respond to prediction-market data—where the "correct" answer is defined by the market consensus at a specific historical date—they identify features that correlate with either "time-aware" reasoning (respecting the historical cutoff) or "look-ahead-biased" reasoning (relying on future knowledge). They then test the causal role of these features by amplifying them during free-form generation tasks, specifically in mergers and acquisitions (M&A) and pharmaceutical forecasting, where the ground truth is known but the model should not have access to it.

The authors find that they can successfully identify and amplify features associated with time-awareness. When these features are boosted during inference, the model exhibits a significant reduction in look-ahead bias, meaning it is less likely to cite future events or post-cutoff knowledge in its reasoning. Crucially, this intervention does not degrade the model's general reasoning capabilities, as measured by performance on standard benchmarks like MMLU. Conversely, the authors were unable to find any "look-ahead-bias" features that, when steered, produced a consistent or useful change in behavior, suggesting that time-awareness is a more robust and manipulatable internal construct.