val_check_interval and check_val_every_n_epoch appear redundant?

[turian]

val_check_interval and check_val_every_n_epoch appear redundant, except val_check_interval can be a float and operate within training epochs.

Is there a difference between the two parameters and can they be used simultaneously? If so, can that be documented clearly?

If not, can check_val_every_n_epoch be marked as deprecated and users pointed towards val_check_interval?

[SkafteNicki]

They are different:

• Use val_check_interval if you want to do validation more than 1 time per training epoch
• Use check_val_every_n_epoch if you want to do validation less often

The serve different scenarios:

• Users will often use val_check_interval when their training data is so large that they are only performing 1 or 2 epochs maybe. Then it is beneficial to do validation more often because your model is changing significantly over a single training epoch
• Users will use check_val_every_n_epoch when either their training data is very small, meaning that the model does not change that much over a single epoch and/or the validation dataset is very large so it may take a long time to validate.

[SkafteNicki]

Is it correct in your proposal, if a user wants to do validation every 1000 step then they would need to work out the fraction
val_check_interval = 1000 / len(dataloader) ?

[turian]

Thinking through this a bit more, my proposal so as not to break backwards compatability would be:

check_val_every_n_epoch can now take a float value.

val_check_interval is deprecated, removed from the documentation, and a warning is issued if it is used.

[SkafteNicki]

There is no way to not break backwards compatability by only having one argument. Your new proposal would remove the possibility to validate after x number of training batches.

[turian]

It just bugs me that val_check_interval has different semantics for float and int, and is confusing.

check_val_every_n_epoch = x allows you to check every x batches. If it's really required to check by number of batches (steps) and we don't want people to divide by the size of data loader, then check_val_every_n_step could be added and only check_val_every_n_epoch or check_val_every_n_step could be non-None

[SkafteNicki]

Fair point, that you find them confusing.
I does seem that we have an old issue that is related to this: #4409.
It seems that we will be keeping val_check_interval and remove check_val_every_n_epoch. Then val_check_interval=2.0 will be interpret as validation every second epoch.