Selecting the best model when validation set is noisy

[adrian-dalessandro]

When I train models for some task, I sometimes find that within a reasonable budget I cannot remove significant noise in a validation set. Sometimes, a low training loss will correlate with a higher validation loss (this shows up often in small datasets and imbalanced regression problems). This makes selecting the best performing epoch difficult during re-training on both the training and validation set. Simply selecting the best performant epoch doesn't work as it may be "unlucky". In practice, what are actually the best strategies for approaching this situation? I saw model averaging mentioned in the playbook. Early stopping may also be an option. What would be reasonable strategy in this scenario?

[varungodbole]

@jmgilmer might have some thoughts on this.