Storing datasets on disk

[jan-swiatek]

Hi!

Since loading single TrackingDataset usually takes about 15 seconds, it would be great if it could be serialized and stored on disk (usefull when processing hundreds of games at once). However it seems that it's not obvious task, because when trying to pickle it I got RecursionError: maximum recursion depth exceeded while pickling an object. I tried both pickle and dill, unfortunately with same result.

Is it possible to optimize this process?

[UnravelSports]

FWIW This infinite recursion occurs because every player object seems to reference every other player object in a loop. This should be addressed at some point, but it might take a bunch of rework @koenvo?

Not sure if you want to store the raw Kloppy objects, but I'd recommend storing the converted tracking data as a parquet file using to_df. But that's not exactly what you're going for.

Any reason for wanting to store the Kloppy objects directly?

[jan-swiatek]

It was just supposed to be part of data loading process - I didn't want to limit the possibilities for further processing. For now I'll probably stick with some dataframe format, but I believe it will be great feature to add 😃

[koenvo]

Pickling the dataset should work in the end. Maybe we can make some small changes to prevent recursion errors.

My idea for serialization to disk would be to use CDF via TrackingDataContainer ( https://github.com/PySport/tracking_data_container/blob/main/RFC.md ). Maybe this is a good opportunity to write a first version of TDC to make this happen.