You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
i believe that is caused by the out of memory. i open the system monitor and found that after running the traincaption.py script, the memory usage continues to increase until it is full, causing an oom error. my computer has 16g memory, is the problem caused by insufficient memory? Or is it because there is a problem in the code that resources are not released in time?
Dufresue
changed the title
multi gpus training error
multi gpus training cause out of memory error
Nov 16, 2023
when i try to run the train_caption.py script like this:
i encountered some errors like this:
![bug2](https://private-user-images.githubusercontent.com/47912661/281243510-bf1e6490-d4dd-45ba-a93b-eae21386b33e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3NTIzNDIsIm5iZiI6MTcyMDc1MjA0MiwicGF0aCI6Ii80NzkxMjY2MS8yODEyNDM1MTAtYmYxZTY0OTAtZDRkZC00NWJhLWE5M2ItZWFlMjEzODZiMzNlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzEyVDAyNDA0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg2ZDc3M2MyMmY4MDIwMzRiNjA2ODE0OTIwNzMwODU4MmQwN2MzZmVhMzNlNGUwNmY1MmE2ZTBiZGI0M2E1NGYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.8ZAza7U9XRBz5o_X2aQwk31By9dLooE3Lr0_K8XssNc)
belows are changes i made in coco_config.yml:
ngpus_per_node:2
world_size:2
batch_size:4
num_workers:2
however,when i set the ngpus_per_node:1 world_size:1,it can run properly.
![微信图片_20231108104933](https://private-user-images.githubusercontent.com/47912661/281244524-12a2701c-91e9-440e-af52-b68cb2e99765.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3NTIzNDIsIm5iZiI6MTcyMDc1MjA0MiwicGF0aCI6Ii80NzkxMjY2MS8yODEyNDQ1MjQtMTJhMjcwMWMtOTFlOS00NDBlLWFmNTItYjY4Y2IyZTk5NzY1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzEyVDAyNDA0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk2MDliZjc5MTg3OGIwZjk3ZjczNTJmOTA2ZmU5MWU0MzRjMDNjM2E0YTQ3ZjgyMTNhMGNlODkwYzdmMTAzYjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.OL3vOYozf5nnxaCJmHkvy-HxR7k2uPaxyEPCqMEY4ro)
anyone can help? thanks a lot!
The text was updated successfully, but these errors were encountered: