The correct way to resume experiment with logger

[ashleve]

From what I understand checkpoints don't save id of the logger run right?
Then what is the correct way to resume training from checkpoint and resume the correct logger run at the same time?

For example, I'm using Wights&Biases logger where you can pass id argument which resumes run experiment:

WandbLogger(id=run_id)

How can I know what is the correct run id without checking it manually?
Is it possible to somehow store it in the checkpoint?
Also, why is checkpoint not storing things like that automatically? Seems like a useful feature.

[manifoldhiker]

Have the same question.

Maybe, ModelCheckpoint should also have an option to save wandb state as well?

[jonashaag]

Same question here.

[jonashaag]

Still an issue, stalebot.

[rlone2018]

I am also using Wights&Biases logger in PyTorch lightening. Due to the limited GPU runtime in COLAB, I have to save my checkpoints and resume them in the next run. The problem is that my wandb graphs show discontinuity and I get separate lines instead of a straight one. Please help me to solve this issue.

[awaelchli]

Try to select global_step on the abcissa, maybe that helps

[awaelchli]

Also, why is checkpoint not storing things like that automatically? Seems like a useful feature.

If you make a feature request on github, I will implement it.

[awaelchli]

You can also save it by overriding the on_save_checkpoint hook and dump the logger.version in there.

[ashleve]

@awaelchli feature request has been here for a while: #5342
I actually thought about implementing it myself but I'm too busy at the moment