Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cuda error( core dump ) #6

Open
xuanson97hg opened this issue Nov 24, 2021 · 3 comments
Open

Cuda error( core dump ) #6

xuanson97hg opened this issue Nov 24, 2021 · 3 comments

Comments

@xuanson97hg
Copy link

when I train with my custom data set if I keep default config.num_classes and config.num_image it work and if I set config.num_classes and config.num_image by my classes and number image of my dataset it error with frame and core dump

@fdbtrs
Copy link
Owner

fdbtrs commented Dec 16, 2021

how many identities dose your training dataset has? you may consider a smaller batch size to fit your model in the GPU

@xuanson97hg
Copy link
Author

how many identities dose your training dataset has? you may consider a smaller batch size to fit your model in the GPU

i try with smaller batch then code run fill minute and core dump again . my dataset has 12k image

@fdbtrs
Copy link
Owner

fdbtrs commented Jan 30, 2022

Resnet 100 with 512 batch requires approximately 4 gpus each of 16GB.
You need to set the batch size based on number of gpu and gpu memory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants