ResearchPod Summary
When applying time-series foundation models (TSFMs) to hourly equity returns, researchers observe an unexpected phenomenon termed "forecast collapse": predictions become nearly flat and exhibit poor cross-sectional ranking, as measured by the information coefficient (IC). Surprisingly, this collapse largely disappears when forecasting trading volume under identical experimental protocols. This paper investigates forecast collapse across TSFMs, twelve deep-learning models, and 97 public benchmark configurations, linking the issue directly to target predictability. The study exposes a critical blind spot in conventional time-series evaluation: per-series metrics can completely hide failures in cross-series structure that are vital for downstream decision-making.
The authors identify two distinct theoretical reasons behind forecast collapse. First, low target predictability inherently limits the amplitude of calibrated point forecasts. When a target has a very low signal-to-noise ratio, calibrated models shrink their predictions toward zero relative to the realized target. Second, standard per-series objectives leave cross-series structure completely unidentified. Because pointwise risks only evaluate individual series against their own targets, forecasts with identical per-series risk can yield drastically different cross-sectional correlations.
These mechanisms establish a strict calibration-ranking tradeoff. Optimizing squared error (MSE) leads to flat predictions with controlled scale but poor ranking. Conversely, directly optimizing cross-sectional correlation improves ranking but can inflate forecast amplitude by more than an order of magnitude. Rescaling a forecast can alter its magnitude, but it cannot recover missing ranking information.
To overcome the calibration-ranking tradeoff, the authors introduce CalibRank, a simple composite objective that jointly optimizes MSE and cross-sectional correlation using a balancing parameter. Evaluated on Finance1K—a new public panel of 1,000 US equities with aligned return and volume targets—CalibRank nearly triples cross-sectional correlation while keeping forecast amplitude close to the true target scale. Furthermore, it consistently improves ranking performance across all tested baseline models.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a study by Shu Wan and colleagues from Arizona State University and the University of Oxford. They're investigating something they call forecast collapse — a puzzling failure mode in modern prediction systems applied to financial markets.
Sam: So what exactly is forecast collapse? What does it look like in practice?
Alex: Imagine you ask a weather app to predict tomorrow's temperature for ten different cities. Instead of giving you different numbers for each city, it just says "72 degrees" for all of them. That's essentially what's happening here. When researchers applied large, sophisticated forecasting models to hourly stock returns, the predictions flattened out into near-zero lines — completely useless for deciding which stocks to buy or sell.
Sam: That seems like a fundamental problem. If everything looks the same to the model, you can't rank anything.
Alex: Exactly. And what makes this interesting is that it's not a bug in one specific model. The authors found it happening across many different systems. So they started asking: what is it about stock return data that causes this, and can we fix it at the training level?
Sam: How did they figure out what was causing it?
Alex: They ran a telling experiment. They kept everything identical — same models, same setup — but changed what they were predicting. When they switched from stock returns to trading volume, the collapse disappeared entirely. The models worked fine.
Sam: Same model, same architecture, but completely different behavior just by changing the target. What does that tell you?
Alex: It tells you the problem lives in the data itself, not the model design. Here's the key idea. Think of any dataset as having two parts: a meaningful pattern you can actually learn from, and random noise you can't predict no matter how smart your model is. Stock returns are almost entirely noise. The genuine predictable signal is extremely weak.
Sam: And when the signal is that weak, what happens during training?
Alex: This is where it gets subtle. When a model is trained to minimize its average error, and the target is mostly unpredictable noise, the mathematically safest strategy is to predict something close to zero for everything. You never make a catastrophically wrong call, but you also never say anything useful. The model learns to be cautiously wrong rather than boldly right.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: So the training process itself is pushing the model toward flatness.
Alex: Right. And there's a second problem layered on top of that. Standard training scores each stock independently — it asks, "how wrong were you about Apple?" and "how wrong were you about Microsoft?" as separate questions. It never asks, "did you correctly rank Apple above Microsoft?"
Sam: So a model could score well on individual errors while completely failing at the thing investors actually care about — which stock will do better than another.
Alex: Precisely. That gap between individual accuracy and cross-sectional ranking is the hidden flaw. You can have a model that looks decent on standard benchmarks and is still useless for building a portfolio.
Sam: So how do the authors propose fixing this?
Alex: They introduce a composite training objective. Think of it like a report card with two grades instead of one. The first grade is the traditional one — how close were your individual predictions to the actual values? The second grade is new — did you correctly rank the stocks relative to each other at each point in time? The model has to do well on both to pass.
Sam: And does adding that ranking grade actually help?
Alex: Across twelve different model architectures, adding the ranking component meaningfully improved the models' ability to correctly order stocks. The individual prediction errors did increase slightly — there's a genuine tradeoff — but the improvement in ranking was substantial, and the error stayed far lower than if you'd optimized for ranking alone without any accuracy anchor.
Sam: So you're finding a middle ground between two competing goals.
Alex: That's a good way to put it. There's a frontier of possible operating points, and practitioners can choose where on that frontier they want to sit depending on what matters more for their specific use case.
Sam: Are there limitations to keep in mind here?
Alex: Several important ones. The empirical results come primarily from one specific financial dataset, so it's not yet clear how broadly these findings generalize to other markets or other types of time-series data. The ranking measure they use — Pearson correlation — captures linear relationships between stocks, but it doesn't capture more complex dependencies, like what happens during a market crash when everything moves together in unusual ways. And the authors are candid that future work will need to address those richer joint behaviors.
Sam: So this is a meaningful step forward, but with real boundaries on what it claims.
Alex: That's a fair summary. The paper identifies a structural reason why sophisticated models fail on low-signal data, and it offers a principled fix that works across a range of architectures. But the authors are careful not to overstate it. The field will likely need to extend these ideas to handle tail risks and more complex market dynamics before this becomes a complete solution.
Sam: It's a good reminder that even the most advanced models are only as useful as the objectives they're trained on.
Alex: Well put. The model learns exactly what you ask it to learn — and if you ask the wrong question, you get the wrong answer, no matter how powerful the system is. Thanks for listening to ResearchPod.