inplace operation Runtime Failure

[JPFrancis]

Hi, I work out of the Michelson Center at USC and I'm trying to run the cbvaegan2D_target.sh script in the /examples/training_scripts folder.

My objective ultimately is to use the model on labeled soft xray tomography data to predict the insulin vesicle label field given membrane, nucleus, and mitochondria labels.

I am running into the following error after the model is initialized. Any help would be great. Thank you!

Traceback (most recent call last):
File "/home/jpfrancis/anaconda3/envs/pytorch_integrated_cell/bin/ic_train_model", line 33, in
sys.exit(load_entry_point('pytorch-integrated-cell', 'console_scripts', 'ic_train_model')())
File "/data/jpfrancis/Development/related_work_code/pytorch_integrated_cell/integrated_cell/bin/train_model.py", line 484, in main
model.train()
File "/data/jpfrancis/Development/related_work_code/pytorch_integrated_cell/integrated_cell/models/base_model.py", line 89, in train
errors, zLatent = self.iteration()
File "/data/jpfrancis/Development/related_work_code/pytorch_integrated_cell/integrated_cell/models/cbvaegan_target2.py", line 185, in iteration
minimaxDecDLoss.mul(self.lambda_decD_loss).backward()
File "/home/jpfrancis/anaconda3/envs/pytorch_integrated_cell/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/jpfrancis/anaconda3/envs/pytorch_integrated_cell/lib/python3.7/site-packages/torch/autograd/init.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512, 15360]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).