Easiest way to plot validation vs. actual data?

[El-Don-Quijote]

Hi, as usual first things first: thank you for your great work and bringing automated time series forecasting to the masses!

I have a rather simple request: I would like to plot the (cross-) validation-forecasts a model (or actually only the selected model) predicts against the actual data. The actual data can be easily accessed with the validation-index of the model, but I am struggling with accessing the validation-runs. Any hints? Greatly appreciated!

Thanks in advance!

[winedarksea]

I only store the aggregated results. With mosaic ensembling, it also saves the point-by-point absolute errors in model.initial_results.full_mae_errors. This is to save memory. If you run a lot of generations and save all the results with a large dataset, you are quickly using a lot of memory if all forecasts are saved.

There is a way to generate those after a model is fit, using the back_forecast function, which is standalone as well as part of the AutoTS class. I would run model.back_forecast(n_splits="auto", tail=model.forecast_length * 3).forecast to get the last three most recent periods. There's also a plot_backforecast that's the same, but generates a plot. The only thing is these are a bit slow as they have to rerun the model for each split, so as per this example, 3 times behind the scenes like 3 validation splits. That is in order to mimic actual result generation, out-of-sample fit, which I feel is better than in-sample fit.

I'm open to suggestions if you think there's a better way to handle this, it's definitely a common use case.

[winedarksea]

Here's what that looks like now. I made some updates which won't be out for a month or two probably...

[El-Don-Quijote]

Hi Colin, thank you. I see, that's good to know and the "tail"-argument makes sense to me. I can't seem to grasp however, why the back_forecast functions needs the "n_splits"-argument? Why do we need to split our dataframe?

From the code:
"n_splits(int): how many pieces to split data into. Pass 2 for fastest, or "auto" for best accuracy"

[winedarksea]

Well what I would recommend is the "auto" which is where it runs however many splits (validations) it takes to cover the entire data period with forecast_length forecast lengths corresponding to how the model would have looked run in the past.
If a different n_splits is passed, it will alter forecast_length to whatever is required to cover the entire data with that many validations. So n_splits really adjusts the forecast_length. The idea is that if you want to see how your model does across the entire history, but you don't want to wait to run hundreds of model runs (say if forecast_length=1 with multiple years of history), it will run a longer forecast length (say 1/4 of your data if n_splits=4) to get a faster, albeit less accurate and representative answer.