The False Security of Static Accuracy: Why Time Series Foundation Models Need Live Evaluation

The Mirage of the Held-Out Test Set

In the rush to deploy time series foundation models (TSFMs), we have allowed ourselves to be seduced by a dangerous fiction: that a single number on a static benchmark suffices to certify readiness for production. Papers routinely report aggregate accuracy on frozen datasets like GIFT-Eval or the Monash Forecasting Repository, treating these scores as proxies for real-world utility. Yet as the authors of "Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting" argue, this protocol measures only cross-sectional generalization, the ability to interpolate across similar series, rather than temporal generalization, the capacity to persist in accuracy as the future unfolds and distributions shift.

The distinction matters profoundly. Static train-test splits, even when carefully constructed, assume that the data generating process remains invariant between training and evaluation periods. They cannot detect when a model has merely memorized seasonal artifacts or when its performance would collapse under the viral traffic spikes, platform migrations, or policy changes that characterize operational environments. Worse, as foundation models scale and training datasets become opaque aggregations of publicly available sources, the risk of test-set contamination grows acute. A model trained on web-scraped temporal data may have already seen what we intend to be its unseen evaluation window, rendering accuracy metrics meaningless.

The Impermanent Protocol: Measuring Robustness, Not Curve Fitting

To address these limitations, the Impermanent benchmark introduces a prequential evaluation framework instantiated on GitHub activity streams. Rather than scoring models on a fixed historical slice, Impermanent operates as a living system. Each day, models must generate forecasts for the next horizon before the ground truth arrives. Only after the events occur are predictions scored and aggregated. This sequential, deployment-faithful loop makes it impossible to cheat through data leakage and forces explicit measurement of how model performance evolves under ongoing distributional shift.

The dataset construction reveals careful attention to non-stationary dynamics. The authors select the 400 most starred repositories and extract four event types: issues opened, pull requests opened, push events, and new stargazers. These series are constructed at four frequencies (hourly with horizon 24, daily with horizon 7, weekly with horizon 4, and monthly with horizon 1), providing a spectrum of temporal granularities. What makes this selection theoretically interesting is not merely the scale but the inherent volatility of the domain. GitHub activity is shaped by software releases, shifting contributor behaviors, changes in platform tooling, and external events such as security vulnerabilities going viral. It is, by design, a system that refuses to stay still.

The authors augment standard error metrics with spectral analysis to characterize these dynamics quantitatively. For each series, they compute the spectral centroid C and normalized spectral entropy H, derived from the real Fast Fourier Transform coefficients. Formally, for a series y with Fourier coefficients X_k and power P_k, the centroid C represents the frequency band containing the majority of the signal's power, while H measures the diffuseness of the spectrum. Higher C indicates faster dynamics; higher H suggests less predictable, more chaotic structure. In their exploratory analysis, the authors demonstrate that different event types occupy distinct regions of this spectral space. Push events might concentrate power at specific frequencies corresponding to CI/CD cycles, while stargazer counts might exhibit more diffuse spectra reflecting viral stochasticity. This diagnostic layer moves beyond aggregate accuracy to reveal why models succeed or fail on specific temporal textures.

From Benchmarks to MLOps Reality

The implications of Impermanent extend beyond academic evaluation protocols into the practical architecture of machine learning operations. Current TSFMs often claim "zero-shot" generalization across domains, yet static benchmarks cannot validate whether these models possess genuine predictive capacity or merely sophisticated interpolation over memorized patterns. When evaluated sequentially on live streams, many models that perform admirably on static splits will likely exhibit catastrophic forgetting or brittleness in the face of concept drift. The spectral heterogeneity of the GitHub data, with its mix of rhythmic and chaotic series, serves as a stress test for claims of robustness.

Consider the practical scenario of forecasting infrastructure load for a popular open source project. A static evaluation might show excellent performance during periods of steady growth, rewarding models that simply extrapolate linear trends. However, when a critical security patch triggers a sudden influx of issues, or when a major conference announcement drives stargazer counts to spike unpredictably, models trained on historical averages may fail systematically. Impermanent captures these regime changes as they occur, providing a measure of sustained performance rather than snapshot accuracy. This aligns with the prequential viewpoint in probability forecasting, which holds that models should be judged by the sequence of predictions they make through time, not by their fit to a held-out past.

The current limitation of the benchmark to univariate series, forecasting one repository-event pair at a time without cross-repository covariates, is both a constraint and an opportunity. While it prevents the evaluation of multivariate dependencies, such as how activity spikes in upstream dependencies propagate downstream, it isolates the core challenge of temporal generalization from the confounding effects of correlation structure. Future iterations that incorporate cross-series attention and external covariates, such as platform updates or global software vulnerability disclosures, will likely prove even more diagnostic.

The Path Forward

Impermanent represents a necessary correction to the field's overreliance on static validation. By shifting the unit of evaluation from the train-test split to the continuous prediction stream, it forces us to confront an uncomfortable truth: most current forecasting models are optimized for a world that does not exist, one where the future resembles the past in easily parameterizable ways. The live evaluation protocol exposes the brittleness that static benchmarks obscure, particularly for foundation models that may have inadvertently memorized test periods during pretraining.

Looking ahead, the success of this benchmark will depend on community adoption and the evolution of drift-aware training methodologies. We need models that degrade gracefully, maintaining calibrated uncertainty estimates as distributions shift, rather than models that memorize past seasons and fail silently when those seasons change. The spectral analysis tools introduced in the paper provide a language for describing these shifts beyond simple error accumulation, potentially enabling the development of adaptive training protocols that respond to changes in spectral entropy or centroid drift.

The broader lesson extends to AI evaluation generally. As foundation models proliferate across domains, we must abandon the comfort of static leaderboards in favor of living benchmarks that respect the arrow of time. Forecasting, by definition, concerns the future. It is past time our evaluation protocols acknowledged this basic fact.