The trainers runs a single validation step after resume (not sanity)

[cdancette]

I am encountering a weird behavior using lightning (I use the lightningCLI as well).

When I resume a training that has failed, with python train.py fit, the trainer runs a single validation step that is not a sanity check (I disabled sanity checks, and also the state of the trainer is RunningStage.VALIDATING).

I run my main training with val_check_interval=0.5.

This is a problem, as it saves in my logs a metric point for this single batch as if it was an evaluation on the full validation set.
I have metrics curve like this: when I resume, there is a wrong point (much higher here than the other ones).

Do you know what could cause this issue and how to get around it ?

[arnaudstiegler]

Seeing a similar behavior with version 2.0.2 and 2.0.7 (latest). Might be worth opening an issue

[jojonki]

I also faced this problem at 2.2.0. I set num_sanity_val_steps=0 at Trainer and use an epoch-end checkpoint.

[jojonki]

I debugged the phenomenon step by step. The issue is that during the restarting status, 0 steps of training and 1 step of validation are performed. To fix this, it is necessary to correct the restart logic, which seems to require a deep understanding of the PL code (I gave up and decided to allow this 1-step validation).

When loading a checkpoint, restarting is set to True.

pytorch-lightning/src/lightning/pytorch/loops/loop.py

Line 84 in 1439da4

self.restarting = True

When training begins, it first checks if restarting is True and if validation is defined. If resuming training, it matches this condition and skips training. The Training Epoch progress bar ends at 0%.

pytorch-lightning/src/lightning/pytorch/loops/training_epoch_loop.py

Lines 199 to 201 in 1439da4

	if self.restarting and self._should_check_val_fx(data_fetcher):
	# skip training and run validation in `on_advance_end`
	return

Next, it enters the validation loop.

pytorch-lightning/src/lightning/pytorch/loops/fit_loop.py

Lines 197 to 200 in 1439da4

	self.setup_data()
	if self.skip:
	return
	self.reset()

In EvaluationLoop's setup_data, it returns immediately, and data_fetcher remains None.

pytorch-lightning/src/lightning/pytorch/loops/evaluation_loop.py

Lines 147 to 148 in 1439da4

	if self._combined_loader is not None and trainer_fn == TrainerFn.FITTING and not self._should_reload_val_dl:
	return

Following the reset, data is "prefetched only once".

pytorch-lightning/src/lightning/pytorch/loops/evaluation_loop.py

Line 219 in 1439da4

data_fetcher = _select_data_fetcher(trainer, trainer.state.stage)

pytorch-lightning/src/lightning/pytorch/loops/fetchers.py

Line 95 in 1439da4

def __init__(self, prefetch_batches: int = 1) -> None:

In EvaluationLoop, self._restarting becomes False, and normal training begins thereafter.

pytorch-lightning/src/lightning/pytorch/loops/fit_loop.py

Line 210 in 1439da4

self._restarting = False

[Youyoun]

Hello (:wave: @cdancette), I am encountering the same issue. A single validation step is performed after a mid-epoch checkpoint is loaded, where the stage is not set as sanity according to Trainer. Any news or issue related to this discussion ?

[korotaS]

Seems that this issue was already present a long ago (#11504) and it was even fixed (#11552), but if we look on current code is seems that the fix is gone...

The fix in version 1.5.10 (lines 534-535):

pytorch-lightning/pytorch_lightning/loops/epoch/training_epoch_loop.py

Lines 530 to 538 in 9ebdc52

	# TODO(@awaelchli): let training/eval loop handle logic around limit_*_batches and val_check_batch
	is_val_check_batch = is_last_batch
	# while restarting with no fault-tolerant, batch_progress.current.ready is -1
	if batch_idx == -1:
	return False
	if isinstance(self.trainer.limit_train_batches, int) and is_infinite_dataset:
	is_val_check_batch = (batch_idx + 1) % self.trainer.limit_train_batches == 0

That same code in the next version 1.6.0 ant later versions doesn't have the fix:

pytorch-lightning/pytorch_lightning/loops/epoch/training_epoch_loop.py

Lines 522 to 528 in 44e3edb

	# TODO(@awaelchli): let training/eval loop handle logic around limit_*_batches and val_check_batch
	is_val_check_batch = is_last_batch
	if isinstance(self.trainer.limit_train_batches, int) and is_infinite_dataset:
	is_val_check_batch = (batch_idx + 1) % self.trainer.limit_train_batches == 0
	elif self.trainer.val_check_batch != float("inf"):
	is_val_check_batch = (batch_idx + 1) % self.trainer.val_check_batch == 0
	return is_val_check_batch

[theodorblackbird]

Facing the same issue, it is particularly annoying when using checkpoint callback with save_top_k since it tracks possibly wrong "best metric".

[shirondru]

I'm also facing this issue

[SeminKim]

Also facing the same issue. I think fixing this correctly is a bit tricky, so I just ended up with dirty workaround:
manually exit train epoch loop with on_validation_model_zero_grad hook, which is called just before val_loop.run() in train_loop.on_advance_end(). Worked for me with Lightning 2.1.3.

    def on_validation_model_zero_grad(self):
        '''
        Small hack to avoid first validation on resume. 
        This will NOT work if the gradient accumulation step should be performed at this point.
        '''
        super().on_validation_model_zero_grad()
        if self.trainer.ckpt_path is not None and getattr(self, '_restarting_skip_val_flag', True):
            self._restarting_skip_val_flag = False
            raise StopIteration

[ethanhe42]

this works for me but somehow messed up global_step counter

[ethanhe42]

here's a better fix without StopIteration

    def on_validation_model_zero_grad(self) -> None:
        '''
        Small hack to avoid first validation on resume. 
        This will NOT work if the gradient accumulation step should be performed at this point.
        '''
        super().on_validation_model_zero_grad()
        if self.trainer.ckpt_path is not None and getattr(self, '_restarting_skip_val_flag', True):
            self.trainer.sanity_checking = True
            self._restarting_skip_val_flag = False

[magehrig]

I also encountered this problem. Is there an github issue open for this? This definitely deserves attention

[ethanhe42]

this is very annoying. deserves more attention

[Youyoun]

I created an issue but wasn't addressed yet: #20288