[FEATURE] Model checkpoint save/load

[rodrigo-arenas]

It would be nice to enable saving and loading models checkpoints, this could help to control training the model in different sessions in case is a large model, as well as keeping a copy of the model in case of some error during the training time

I open this issue for contributors

This Issue requests the following features:

Describe the solution you'd expect

• Enable saving model checkpoints as a callback named ModelCheckpoint that takes as an argument the location to save the model
• The checkpoints should save the training status and the logbook object, you can make use of the already implemented class LogbookSaver
• Implement save and load methods in GASearchCV and GAFeatureSelectionCV
• When calling the fit method, it should resume the training where it was left by default
• Enable an option to start the training again (from generation 0) but with starting point (i.e hyperparameters or features) the best ones found so far in the saved model

Additional context
You can check TensorFlow save and load weights methods as an inspiration

[cpparnell]

I would like to work on this, is that alright?

[rodrigo-arenas]

Hi @cpparnell sure, you can work on it, thanks

[cpparnell]

@rodrigo-arenas I have some questions about the last two bullet points:

• When calling the fit method, it should resume the training where it was left by default
• Enable an option to start the training again (from generation 0) but with starting point (i.e hyperparameters or features) the best ones found so far in the saved model

Should there be an option provided to the fit method to start training again? Or should the functionality described above be implemented within the ModelCheckpoint callback?

Do you mean that I should be able to use the load method to load from the checkpoint path, similar to what is described in TensorFlow's ModelCheckpoint?

[rodrigo-arenas]

Hi, there is already a version working with some of the specifications made by @The-Blitz at #158, however, we still need to work in the feature:

• When calling the fit method, it should resume the training where it was left by default
• Enable an option to start the training again (from generation 0) but with a starting point (i.e hyperparameters or features) the best ones found so far in the saved model

The second one should be possible by using the recently added parameter warm_start_configs

[The-Blitz]

@rodrigo-arenas I just noticed the PR was pushed haha. I thought there were pending errors on some tests which I could not see locally. Anyway I will try to check the missing points when possible, if you don't mind.