TopoCast: A Topological Fidelity Framework for Evaluating Transformer-Based Time Series Forecasting

Deep learning models, particularly Transformers, have revolutionized time series forecasting (TSF). However, their performance is almost exclusively measured using pointwise metrics like Mean Squared Error (MSE) or Mean Absolute Error (MAE). These metrics quantify numerical accuracy at specific time steps but are fundamentally blind to the underlying structural dynamics of the signal. A forecast can achieve a low MSE while completely failing to preserve critical features like oscillatory patterns, phase alignment, or recurrent dynamics. This creates a significant gap between numerical accuracy and structural fidelity, leading to models that appear high-performing while failing to capture the true behavior of the dynamical system.

TopoCast addresses this limitation by applying Topological Data Analysis (TDA) to forecast evaluation. Instead of comparing raw values, TopoCast reconstructs the phase space of both the ground truth and the forecast using Takens delay embedding. This process lifts the one-dimensional signal into a higher-dimensional space where its geometric structure—such as loops representing periodic cycles—becomes explicit.

By applying persistent homology to these embeddings, the framework extracts topological descriptors that characterize the signal's intrinsic dynamics. These are aggregated into a Topological Fidelity Score (TFS). To specifically address temporal errors, the authors introduce a novel metric called dominant cycle overlap. This metric maps the topological features back to the original time domain, allowing researchers to verify if the model's predicted oscillations occur at the correct time points. The combination of these measures results in the Localized Topological Fidelity Score (LTFS).

Experimental results across five Transformer architectures and three real-world datasets show that models with nearly identical MSE scores can exhibit vastly different structural fidelity profiles. TopoCast reveals that conventional metrics often mask significant structural degradation, such as temporal phase shifts or the suppression of high-frequency components. By providing a phase-aware, topology-based evaluation, TopoCast offers a more rigorous way to assess whether a model truly understands the temporal structure of the data, which is essential for high-stakes applications in fields like finance, energy, and healthcare.