Multi-variate Multi-length sequences

[erezinman]

Hi,
Thank you for the wonderful package. Can the platform train on multiple multivariate/univariate-with-covariates sequences that are of different lengths?

For example:

seq1 = pd.DataFrame({
     'time': pd.to_datetime(['2001', '2002', '2003']), 
     'col1': [1, 2, 3], 
     'col2': [4, 5, 6]
})
seq2 = pd.DataFrame({
     'time': pd.to_datetime(['2004', '2005', '2006', '2007']), 
     'col1': [1, 2, 3, 4], 
     'col2': [1, 2, 3, 4]
})

If that's possible, can you please also add this case to the tutorial?

Thanks!

[winedarksea]

Thanks, I don't quite understand. Or perhaps I have an over-active imagination.

I see at least three ways of interpreting this:

1. Sequence 1 is only provided as training data, no additional forecasts are desired
2. Sequence 1 is expecting forecasts starting in 2004 while Sequence 2 is expecting forecasts starting in 2008
3. Sequence 1 is actually the same series as Sequence 2, but we just aren't telling the model that (because we have 1000's of series and don't have time to manually align them).

any of these close?

[erezinman]

I meant that they both are (for the sake of the example) train-sequences. In case that I just want my train-set to be comprised of those two sequences (for the sake of the example, say I want a window of size 2).

(So I guess that's option 1?)

To be more concrete, and hopefully, to better explain my intent, my data is a univariate-time series with both past-only and future covariates. The data itself is sampled uniformly, but has "dead" time-ranges that the model mustn't train on, so the data is divided into multiple multidimensional sequences with variable (non-uniform) lengths.

[erezinman]

I'll look at it, thanks. This is actually the one dataset that I didn't look at because I didn't have the prerequisites installed.

[winedarksea]

I think it might help if I explain how it would handle the data if passed in together.
AutoTS would by default construct a dataset like this:

seq12 = pd.DataFrame({, 
     'col1': [1, 2, 3, np. nan, np. nan, np. nan, np. nan], 
     'col2': [4, 5, 6, np. nan, np. nan, np. nan, np. nan],
     'col3': [np. nan, np. nan, np. nan, 1, 2, 3, 4], 
     'col4': [np. nan, np. nan, np. nan, 1, 2, 3, 4]
}, index = pd.to_datetime(['2001', '2002', '2003', '2004', '2005', '2006', '2007']))

it would then try various fillna strategies to deal with those NaN to most accurately minimize the error in cross-validation... which wouldn't work well here because there isn't enough data for cross validation. There's a prefillna=0 option which would allow those missing values to be treated as something else, like 0.

Also another way for dealing with covariates is passing in the lagged values as regressors. See the production_example.py and it's use of create_regressor() for one example of that.

[erezinman]

Thank you for your help! I'll give it a try

[winedarksea]

It occurs to me that passing weights to the .fit() might be something you want too. It looks something like this:

weights = {
    'col1': 1
    'col2': 10
}

basically allowing you to say, "hey model, I'm giving you all these series as information, but really care far more about col2"