Shuffle buffer filled

[berton820]

command: python run_bert_squadv2.py
GPU: P40
CPU memory: 126G

log:
2019-07-28 01:28:56.654470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-07-28 01:28:56.866822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21139 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-07-28 01:29:01.610923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-07-28 01:29:01.611087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-28 01:29:01.611125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-07-28 01:29:01.611143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-07-28 01:29:01.792284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21139 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-07-28 01:29:04.056203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-07-28 01:29:04.056312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-28 01:29:04.056346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-07-28 01:29:04.056365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-07-28 01:29:04.057344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21139 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-07-28 01:29:49.138936: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 7971 of 130497
2019-07-28 01:29:59.139110: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 15820 of 130497
2019-07-28 01:30:09.139348: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 23963 of 130497
2019-07-28 01:30:19.138907: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 31963 of 130497
2019-07-28 01:30:29.139424: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 39830 of 130497
2019-07-28 01:30:39.139672: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 48245 of 130497
2019-07-28 01:30:49.138921: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 56364 of 130497
2019-07-28 01:30:59.139422: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 64376 of 130497
2019-07-28 01:31:09.139234: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 72092 of 130497
2019-07-28 01:31:19.139490: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 79574 of 130497
2019-07-28 01:31:29.139453: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 87148 of 130497
2019-07-28 01:31:39.139663: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 94795 of 130497
2019-07-28 01:31:49.139455: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 102515 of 130497
2019-07-28 01:31:59.140305: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 110001 of 130497
2019-07-28 01:32:09.138877: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 117752 of 130497
2019-07-28 01:32:19.139864: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 125678 of 130497
2019-07-28 01:32:25.347410: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:136] Shuffle buffer filled.
~

[zww847204326]

I have the same trouble, do you solve it?

[berton820]

I have the same trouble, do you solve it?

no