How can I obtain train/validation % of accuracy of the selected model

[akampianakis]

As plenty of stakeholders understand the capability of a model based on the % accuracy (e.g Mean Absolute Percent Error (MAPE) =100 * (ABS (Actual – Forecast)/Actual) )how can we eventually obtain such metrics specifically for training/validation segments of the best model?
Thanks

[winedarksea]

@akampianakis since model objects are not saved, it's difficult to add custom metrics. AutoTS used to save the model predictions for each round into memory, which would allow you to retroactively calculate any metric you wanted, but it took up a huge amount of memory and added no value to most users. If this is a feature you want, create a new Issue add parameter to retain all training predictions in memory or something to that effect.
That said, you can access a bunch of existing metrics for every model quite easily. I don't have MAPE but I do have sMAPE which is better for model optimization. To view that for the training/validation segments:

where model is a trained AutoTS object...

model_results = model.results()

In this table you'll see one record per model per validation (so if you have num_validations=2 there would be 3 records if it was chosen for validation, if it wasn't good enough to make it to validation, there will be only one record from initial run) and models are identified by an ID (a hash of params) as well as parameters, and a validation column to identity which round that accuracy is from.

let me know if this helps, thanks!

[akampianakis]

Perfect, sounds enough for me, but (and correct me if i'm totally wrong, I definitely don't have your depth of knowledge.!) in order to make an understandable to the ultra simple user how good is the model (right now I am building this for accounting documents processing..), is it possible to get the value forecaster by the model during the validation process, compare it with the actual one and create such a metric? I mean something under this logic:
the model selected forecasted for month/day 1: 47 but the actual was 50 (94% accuracy)
the model selected forecasted for month/day 2: 49 but the actual was 48 (98% accuracy)
etc etc etc, so that we can have the ability to show to the everyday user what's the model worth against previously known data.
Thanks in advance, keep up this amazing work, really excited for the next version!!

[winedarksea]

Thanks @akampianakis you brought up a really good point here! I built this with the expectation users would dig into the results table, but its a great idea to make this more accessible, with the ongoing accuracy more apparent during training and easier to access for users.
#62 created to add this at some point...

[akampianakis]

That would make it totally awesome i think , as it will give tacit results for the non-educated users/managers, that would defy the results. Hope we see it soon!Thanks for your amazing work keep on rolling!