ResourceExhaustedError: OOM when allocating tensor of shape

[galoisgroupcn]

Hi,

As I run 'bash experiments/train_can_paper.sh' and 'bash experiments/eval_can_paper.sh' with the provided code, I always get the same error below:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor of shape [5,5,1024,1024] and type float
[[Node: generator/g_h1/w/Adam/Initializer/zeros = Const_class=["loc:@generator/g_h1/w"], dtype=DT_FLOAT, value=Tensor<type: float shape: [5,5,1024,1024] values: [[[0 0 0]]]...>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Does anyone encounter similar errors? Should I load WikiArt in batches instead of reading the whole thing all at once? How may I modify the code? Thank you!

FYI, I am running Ubuntu 18.04 with Nvidia GTX 1050 and Driver Version 390.116

• In train_can_paper.sh, I tried to change the "batch_size" from '16' to '4' & '1', but it still came out with the same error.

Thank you very much!

@philkuz @phillip-kravtsov @ruotianluo @lanmcmill