Error while running DEMO

[PatrykKotlowski]

Hello,
Thank you for sharing your code!
I've got issue while running DEMO. When I run command:
python main.py --AU=1 --fold=0 --GPU=0 --OF Horizontal --DEMO=Demo --mode_data=normal --pretrained_model /home/patryk/git/AU01.pth

I've got this output messages:
Namespace(AU='1', DELETE=False, DEMO='Demo', GPU='0', HYDRA=False, OF=True, OF_option='Horizontal', SHOW_MODEL=False, TEST_PTH=False, TEST_TXT=False, batch_size=118, beta1=0.5, beta2=0.999, dataset='BP4D', finetuning='emotionnet', fold='0', image_size=224, log_path='./snapshot/logs/BP4D/normal/fold_0/AU01/OF_Horizontal/emotionnet', log_step=2000, lr=0.0001, metadata_path='./data/BP4D/normal/fold_0/AU01', mode='train', mode_data='normal', model_save_path='./snapshot/models/BP4D/normal/fold_0/AU01/OF_Horizontal/emotionnet', num_epochs=12, num_epochs_decay=13, num_workers=4, pretrained_model='/home/patryk/git/AU01.pth', results_path='./snapshot/results', stop_training=2, test_model='', use_tensorboard=False, xlsfile='./snapshot/results/normal/emotionnet.xlsx')
[!!] loaded trained model: /home/patryk/git/AU01.pth!
Killed

It happend multiple times but once my pycharm was closed and there was error:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5d47cda000, 16384, 0) failed; error='Not enough space' (errno=12)
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/patryk/java_error_in_PYCHARM_2427.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5d08d63000, 16384, 0) failed; error='Not enough space' (errno=12)
[thread 8672 also had an error]

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f5c95fde994]

Is this related to operating system? What OS did you use?
I'm using C python 2.7 and Ubuntu 16.04.

Thanks!

[affromero]

Hello, it seems a problem with your memory system. I used Ubuntu 16 as well. Were you able to solve it?

[PatrykKotlowski]

I'm still debugging :(
Could you tell me what CUDA version you're using?

[affromero]

Mmhh, it is hard to say because it is been a while, but my best guess would be 8.0. However, it does not seem to be the case of a Cuda related problem.

[PatrykKotlowski]

Yes, you're right. I checked top output and it seems like loading pretrained model takes about 22GB of RAM. After that it keeps growing and it takes all 32GB from my machine.
I didn't do any other changes than changing data paths. Is this normal that this script takes such amount of RAM?

What hardware did you use?

Thanks for helping btw I'm very grateful :)

[affromero]

It is weird, as long as you have the models loaded in the GPU, you should not have such CPU RAM consumption. I ran these experiments in a server with about 128GB of RAM, so I never saw anything like it ^^.