Out of memory in evaluation after train epochs

[caide199212]

When reproducing the REDS datasets, I reduced the num_input_frames=20 to fit the GPU memory, so the training is fine, 8G is used for every 16G GPU.

But when it's evaluating, the GPU consumed by the training iteration seems not released. So the evaluation encounters the OOM as
RuntimeError: CUDA out of memory. Tried to allocate 436.00 MiB (GPU 1; 15.90 GiB total capacity; 10.98 GiB already allocated; 189.81 MiB free; 14.78 GiB reserved in total by PyTorch)

The config file for the data is as below:
data = dict( workers_per_gpu=4, train_dataloader=dict(samples_per_gpu=1, drop_last=True), test_dataloader=dict(samples_per_gpu=1, workers_per_gpu=1),

BTW, similar problem was reported in mmdet but no feasible solutions was found.

[ericzw]

I don't know how to solve this problem.
Maybe you can skip the evaluation during the training process.

[caide199212]

Good idea, thanks.

[sunyclj]

I don't know how to solve this problem.
Maybe you can skip the evaluation during the training process.

when setting code to evaluation = dict(interval=1000000, save_image=False, gpu_collect=True)，the same error occurs.