Possible to resume a finetune run?

[dillfrescott]

Like load from a finetune checkpoint and resume training.

[Pur1zumu]

Update: Code committed to 5a2b672, but not yet tested as I'm working on the whisper branch.
Lightning's fit() method includes a ckpt_path parameter, so theoretically, we only need to set save_weight_only=False and provide ckpt_path and wandb_resume_id to resume training.
I'll merge to master after successful testing.

[Pur1zumu]

Update: the code has been merged into the master branch. Although training can be resumed, for some reason the resumed trainer no longer performs evaluation according to the eval_step in the config. I haven't found the cause yet.

[dillfrescott]

Okay! Thank you!