CUDA out of memory

[Wkzlike]

I followed the environment configuration, but encountered an out-of-GPU-memory error：
ngpus_per_node: 2
2024-06-14 10:13:04
Using GPU: 1 for training
Using GPU: 0 for training
Traceback (most recent call last):
File "/data/run01/scz0rja/server/ADL-main/train_DDP.py", line 319, in
main()
File "/data/run01/scz0rja/server/ADL-main/train_DDP.py", line 100, in main
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
File "/HOME/scz0rja/run/envs/adl/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/HOME/scz0rja/run/envs/adl/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
while not context.join():
File "/HOME/scz0rja/run/envs/adl/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/HOME/scz0rja/run/envs/adl/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/data/run01/scz0rja/server/ADL-main/train_DDP.py", line 198, in main_worker
loss = train(model,optimizer,sample['left'],sample['right'],sample['disparity'],args,gpu)
File "/data/run01/scz0rja/server/ADL-main/train_DDP.py", line 282, in train
+ 0.6 * loss_func(output2,disp_true,mask,args.maxdisp+1)
File "/data/run01/scz0rja/server/ADL-main/losses/gt_distribution.py", line 78, in Adaptive_Multi_Modal_Cross_Entropy_Loss
GT = (GT * w_cluster).sum(dim=1, keepdim=False)
RuntimeError: CUDA out of memory. Tried to allocate 2.15 GiB (GPU 0; 23.70 GiB total capacity; 17.49 GiB already allocated; 1.69 GiB free; 20.19 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[xxxupeng]

Maybe you run the GANet? If so, you need to reduce the batch size. The default configurations are for the PSMNet.

[Wkzlike]

Maybe you run the GANet? If so, you need to reduce the batch size. The default configurations are for the PSMNet.

Thank you very much for your timely response, I have resolved this issue.