Exceptions. 1. TypeError: can't pickle Environment objects 2. EOFError: Ran out of input

[oschakravarthi]

Hi,
I am getting these two exceptions when I train. Can you please help me in fixing them?
Thanks in advance.

D:\Professional\GitHub\CubiCasa5k>python train.py --batch-size=1
D:\Professional\GitHub\CubiCasa5k\floortrans\models\hg_furukawa_original.py:227: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if y.shape != x.shape:
Traceback (most recent call last):
File "train.py", line 416, in
train(args, log_dir, writer, logger)
File "train.py", line 147, in train
for i, samples in tqdm(enumerate(trainloader), total=len(trainloader),
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataloader.py", line 819, in iter
return _DataLoaderIter(self)
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\utils\data\dataloader.py", line 560, in init
w.start()
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in init
reduction.dump(process_obj, to_child)
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle Environment objects

D:\Professional\GitHub\CubiCasa5k>Traceback (most recent call last):
File "", line 1, in
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

[angie-getair]

(Edit 24.07.2020)
We re-created the LMDB database and everything is fine now.

---

(Edit 21.07.2020)
Update Pytorch to 1.1.0 solves the error THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument

How ever the remaining error still happens.

---

Hello,
I face similar problem. Appreciate any helps or advice. Thanks in advance.

ubuntu@ip-172-31-43-245:~/project$ python3 train.py 
/home/ubuntu/project/floortrans/models/hg_furukawa_original.py:227: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if y.shape != x.shape:
/home/ubuntu/.local/lib/python3.6/site-packages/torch/onnx/utils.py:586: UserWarning: ONNX export failed on ATen operator copy because torch.onnx.symbolic.copy does not exist
  .format(op_name, op_name))
  0%|                                                   | 0/400 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train.py", line 417, in <module>
    train(args, log_dir, writer, logger)
  File "train.py", line 188, in train
    for i_val, samples_val in tqdm(enumerate(valloader), total=len(valloader), ncols=80, leave=False):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/tqdm/_tqdm.py", line 1022, in __iter__
    for obj in iterable:
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ubuntu/project/floortrans/loaders/svg_loader.py", line 42, in __getitem__
    sample = self.get_data(index)
  File "/home/ubuntu/project/floortrans/loaders/svg_loader.py", line 92, in get_lmdb
    sample = pickle.loads(data)
TypeError: a bytes-like object is required, not 'NoneType'

Environment:

• NVIDIA Tesla T4, Driver Version: 450.51.05
• Ubuntu 18.04
• Cuda 9.1, V9.1.85
• Python 3.6.9

asn1crypto==0.24.0
attrs==19.1.0
Automat==0.6.0
backcall==0.1.0
bleach==3.1.0
blinker==1.4
certifi==2018.10.15
cffi==1.11.5
chardet==3.0.4
click==6.7
cloud-init==19.4
cloudpickle==0.8.0
colorama==0.3.7
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
cryptography==2.3.1
cycler==0.10.0
Cython==0.29.6
dask==1.1.4
decorator==4.4.0
defusedxml==0.5.0
distro-info===0.18ubuntu0.18.04.1
ec2-hibinit-agent==1.0.0
entrypoints==0.3
future==0.18.2
graphviz==0.8.4
h5py==2.8.0
h5py-cache==1.0
hibagent==1.0.1
httplib2==0.9.2
hyperlink==17.3.1
icc-rt==2020.0.133
idna==2.7
incremental==16.10.1
intel-numpy==1.15.1
intel-openmp==2020.0.133
ipykernel==5.1.0
ipython==7.3.0
ipython-genutils==0.2.0
jedi==0.13.3
Jinja2==2.10
joblib==0.13.2
jsonpatch==1.16
jsonpointer==1.10
jsonschema==3.0.1
jupyter-client==5.2.4
jupyter-core==4.4.0
jupyterlab==0.35.4
jupyterlab-server==0.2.0
keyring==10.6.0
keyrings.alt==3.0
kiwisolver==1.0.1
language-selector==0.1
lmdb==0.94
MarkupSafe==1.1.1
matplotlib==3.0.3
mistune==0.8.4
mkl==2019.0
mkl-fft==1.0.6
mkl-random==1.0.1
nbconvert==5.4.1
nbformat==4.4.0
netifaces==0.10.4
networkx==2.2
notebook==5.7.6
numpy==1.15.4
oauthlib==2.0.6
olefile==0.46
opencv-python==3.1.0.5
PAM==0.4.2
pandas==0.24.2
pandocfilters==1.4.2
parso==0.3.4
pexpect==4.6.0
pickleshare==0.7.5
Pillow==7.2.0
prometheus-client==0.6.0
prompt-toolkit==2.0.9
protobuf==3.7.0
ptyprocess==0.6.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.19
pycrypto==2.6.1
Pygments==2.3.1
pygobject==3.26.1
PyJWT==1.5.3
pyOpenSSL==18.0.0
pyparsing==2.3.1
pyrsistent==0.14.11
pyserial==3.4
PySocks==1.6.8
python-apt==1.6.5+ubuntu0.3
python-dateutil==2.8.0
python-debian==0.1.32
pytz==2018.9
PyWavelets==1.0.2
pyxdg==0.25
PyYAML==3.12
pyzmq==17.1.2
requests==2.20.1
requests-unixsocket==0.1.5
scikit-image==0.14.2
scipy==1.1.0
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
Send2Trash==1.5.0
service-identity==16.0.0
Shapely==1.6.4.post2
six==1.15.0
ssh-import-id==5.7
svgpathtools==1.3.3
svgwrite==1.2.1
systemd-python==234
tbb==2019.0
tbb4py==2019.0
tensorboardX==1.6
terminado==0.8.1
testpath==0.4.2
toolz==0.9.0
torch==1.1.0
torchfile==0.1.0
torchnet==0.0.4
torchvision==0.2.1
tornado==5.1.1
tqdm==4.31.1
traitlets==4.3.2
Twisted==17.9.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.24.1
visdom==0.1.8.5
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.54.0
xkit==0.0.0
zope.interface==4.3.2