Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Darcy 2D l2.backward() error #6

Open
saki606 opened this issue Oct 4, 2022 · 0 comments
Open

Darcy 2D l2.backward() error #6

saki606 opened this issue Oct 4, 2022 · 0 comments

Comments

@saki606
Copy link

saki606 commented Oct 4, 2022

Hoping someone can clear up the error below.
Tryting to run MGKN_general_darcy2d.py and encounter the following error during Training (see below).
Seems like a variable is being overwritten (in_place?) somewhere . . . can't figure out exactly where.
Tried variations of adding variable.clone() in the NN forward feed process to see if it helps, but nothing so far.


preprocessing finished, time used: 257.48266062999755
3 [0, 400, 500, 525]
Traceback (most recent call last):
File "MGKN_general_darcy2d.py", line 280, in
l2.backward()
File "/home//projects/dir_env/env_pytorch/lib/python3.8/site-packages/torch/_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/
/projects/dir_env/env_pytorch/lib/python3.8/site-packages/torch/autograd/init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [525, 64]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

adding torch.autograd.set_detect_anomaly(True) produces the following message:


RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [525, 64]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant