Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible version mismatch #15

Open
csbotos opened this issue Feb 20, 2018 · 4 comments
Open

Possible version mismatch #15

csbotos opened this issue Feb 20, 2018 · 4 comments

Comments

@csbotos
Copy link

csbotos commented Feb 20, 2018 •
edited
Loading 

python reduce_model.py --model-input face2face-model --model-output face2face-reduced-model
/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
2018-02-20 10:39:09.133747: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
2018-02-20 10:39:09.278103: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-02-20 10:39:09.278400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 9.97GiB
2018-02-20 10:39:09.278420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-02-20 10:39:09.484563: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_7/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.484723: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_6/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.484944: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_2/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.485069: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_8/conv/filter not found in checkpoint
2018-02-20 10:39:09.485348: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_2/deconv/filter not found in checkpoint
2018-02-20 10:39:09.485935: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_3/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.485993: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_6/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.486155: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_8/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.486223: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_7/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.486601: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_7/deconv/filter not found in checkpoint
2018-02-20 10:39:09.487070: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_6/deconv/filter not found in checkpoint
2018-02-20 10:39:09.487313: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_2/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.487331: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_8/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.487411: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_8/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.487513: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_5/deconv/filter not found in checkpoint
2018-02-20 10:39:09.487556: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_1/deconv/filter not found in checkpoint
2018-02-20 10:39:09.487598: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_3/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.488191: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_3/deconv/filter not found in checkpoint
2018-02-20 10:39:09.488352: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_8/deconv/filter not found in checkpoint
2018-02-20 10:39:09.488652: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_8/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.488887: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_4/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.488939: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/conv/filter not found in checkpoint
2018-02-20 10:39:09.489140: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_5/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.489250: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_4/deconv/filter not found in checkpoint
2018-02-20 10:39:09.489578: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_2/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.489791: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_4/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.490046: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_2/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.490176: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_1/conv/filter not found in checkpoint
2018-02-20 10:39:09.490245: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.490715: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_2/conv/filter not found in checkpoint
2018-02-20 10:39:09.490881: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_3/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.490945: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_4/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.490986: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_3/conv/filter not found in checkpoint
2018-02-20 10:39:09.491020: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_3/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.491152: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/decoder_5/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.491998: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_4/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.492526: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_5/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.492707: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_6/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.492769: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_4/conv/filter not found in checkpoint
2018-02-20 10:39:09.492810: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_5/conv/filter not found in checkpoint
2018-02-20 10:39:09.492883: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_6/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.492923: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_7/batchnorm/offset not found in checkpoint
2018-02-20 10:39:09.492962: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_5/batchnorm/scale not found in checkpoint
2018-02-20 10:39:09.492991: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key generator/encoder_6/conv/filter not found in checkpoint
Traceback (most recent call last):
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
    return fn(*args)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
    status, run_metadata)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key generator/decoder_7/batchnorm/offset not found in checkpoint
	 [[Node: save/RestoreV2_16 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_16/tensor_names, save/RestoreV2_16/shape_and_slices)]]
	 [[Node: save/RestoreV2_38/_53 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_144_save/RestoreV2_38", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "reduce_model.py", line 215, in <module>
    saver.restore(sess, checkpoint)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1666, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key generator/decoder_7/batchnorm/offset not found in checkpoint
	 [[Node: save/RestoreV2_16 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_16/tensor_names, save/RestoreV2_16/shape_and_slices)]]
	 [[Node: save/RestoreV2_38/_53 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_144_save/RestoreV2_38", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'save/RestoreV2_16', defined at:
  File "reduce_model.py", line 213, in <module>
    saver = tf.train.Saver()
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1218, in __init__
    self.build()
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1227, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1263, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 751, in _build_internal
    restore_sequentially, reshape)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 427, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 267, in restore_op
    [spec.tensor.dtype])[0])
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1021, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/home/csbotos/.virtualenvs/cv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key generator/decoder_7/batchnorm/offset not found in checkpoint
	 [[Node: save/RestoreV2_16 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_16/tensor_names, save/RestoreV2_16/shape_and_slices)]]
	 [[Node: save/RestoreV2_38/_53 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_144_save/RestoreV2_38", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
@sachinruk
Copy link

Hi,

Just my 2cents here but as far as I'm aware pix2pix was trained using TF 0.12.1 whereas in this repo TF 1.2.1 was used. So it's something to do with how the checkpoints were created. Hopefully Dai Train comments on how this was remedied.

I got exactly the same errors and tried changing the environment.yml file but no luck so far. Although to be fair I did train it on Floydhub using TF 1.3.0 so this might be a source of the error as well.

Cheers,
Sachin

@slothlysage
Copy link 

2018-03-11 21:36:12.823068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 5.94GiB freeMemory: 5.40GiB
2018-03-11 21:36:12.823081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1306] Adding visible gpu devices: 0
2018-03-11 21:36:12.999525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:987] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5177 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-03-11 21:36:13.147256: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_2/deconv/filter not found in checkpoint
2018-03-11 21:36:13.147256: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_3/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.147310: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_8/conv/filter not found in checkpoint
2018-03-11 21:36:13.147416: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_3/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.147625: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_1/deconv/filter not found in checkpoint
2018-03-11 21:36:13.147694: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_2/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.148157: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_8/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.148196: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_3/deconv/filter not found in checkpoint
2018-03-11 21:36:13.148248: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_4/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.148342: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_4/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.148489: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_2/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.148524: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_4/deconv/filter not found in checkpoint
2018-03-11 21:36:13.148692: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_5/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.148837: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_8/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.149567: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_5/deconv/filter not found in checkpoint
2018-03-11 21:36:13.149594: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_5/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.149629: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_7/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.149675: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_7/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.149722: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_6/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.149749: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_6/deconv/filter not found in checkpoint
2018-03-11 21:36:13.149787: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_6/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.150169: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_7/conv/filter not found in checkpoint
2018-03-11 21:36:13.150419: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_1/conv/filter not found in checkpoint
2018-03-11 21:36:13.150769: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_2/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.150846: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_8/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.151103: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_8/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.151145: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_2/conv/filter not found in checkpoint
2018-03-11 21:36:13.151265: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_2/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.151311: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_7/deconv/filter not found in checkpoint
2018-03-11 21:36:13.151668: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_3/conv/filter not found in checkpoint
2018-03-11 21:36:13.152074: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_4/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.152223: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/decoder_8/deconv/filter not found in checkpoint
2018-03-11 21:36:13.152370: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_4/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.152442: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_3/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.152476: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_7/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.152833: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_4/conv/filter not found in checkpoint
2018-03-11 21:36:13.152962: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_5/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.153082: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_5/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.153111: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_3/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.153251: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_5/conv/filter not found in checkpoint
2018-03-11 21:36:13.153428: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_6/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.153520: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_6/batchnorm/scale not found in checkpoint
2018-03-11 21:36:13.153635: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_7/batchnorm/offset not found in checkpoint
2018-03-11 21:36:13.153690: W tensorflow/core/framework/op_kernel.cc:1208] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key generator/encoder_6/conv/filter not found in checkpoint
/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1329, in _run_fn
    status, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key generator/decoder_2/deconv/filter not found in checkpoint
	 [[Node: save/RestoreV2_3 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_3/tensor_names, save/RestoreV2_3/shape_and_slices)]]
	 [[Node: save/RestoreV2_16/_35 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_126_save/RestoreV2_16", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "reduce_model.py", line 215, in <module>
    saver.restore(sess, checkpoint)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1755, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key generator/decoder_2/deconv/filter not found in checkpoint
	 [[Node: save/RestoreV2_3 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_3/tensor_names, save/RestoreV2_3/shape_and_slices)]]
	 [[Node: save/RestoreV2_16/_35 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_126_save/RestoreV2_16", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'save/RestoreV2_3', defined at:
  File "reduce_model.py", line 213, in <module>
    saver = tf.train.Saver()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1288, in __init__
    self.build()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1297, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1334, in _build
    build_save=build_save, build_restore=build_restore)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
    restore_sequentially, reshape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 448, in _AddRestoreOps
    restore_sequentially)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 269, in bulk_restore
    self.restore_op(filename_tensor, saveable, preferred_shard))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 296, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 1029, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3267, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key generator/decoder_2/deconv/filter not found in checkpoint
	 [[Node: save/RestoreV2_3 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2_3/tensor_names, save/RestoreV2_3/shape_and_slices)]]
	 [[Node: save/RestoreV2_16/_35 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_126_save/RestoreV2_16", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]```

I had about the same error, using a custom built tensorflow for my computer, tf version 1.5.0-rc1. I was hoping there would be some resolution beyond having 2 tensorflow versions... Currently, pix2pix was able to build the model, but going back to face2face to run the reduce_model.py is my bottle neck. Going to try to figure out if I can hard code some solutions in the script, or manually add what is missing to the checkpoint in the model.

@vivekmathema
Copy link

same problem here. no luck on tf-gpu 1.5.1

@vivekmathema
Copy link

Thsi worked for me... as sugegsted elsewhere

=
The issue is solved by using an older version of affinelayer/pix2pix-tensorflow.

git clone https://github.com/affinelayer/pix2pix-tensorflow
cd pix2pix-tensorflow

Reset to april version

git reset --hard d6f8e4ce00a1fd7a96a72ed17366bfcb207882c7

And then retrain your model and everything should work fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sachinruk @slothlysage @csbotos @vivekmathema