tf_data_bottleneck_analysis hangs forever upon tf.data profling

[leckie-ponyai]

I was using tensorflow profiler to profile tf.data pipeline bottleneck. So I wrote a benchmark program which has the following code pieces:

train_set = TrainDataset(...) # create tf.data pipeline
train_data_op = train_set.get_next()
with tf.profiler.experimental.Profile(FLAGS.summary_dir):
    for _ in range(1000):
        _ = sess.run(train_data_op)

which ran remotely and I downloaded the generated profile data to local.

Then I got the following issue:

(base) ➜ tensorboard --logdir ~/work/smart_agent/runs/profile --load_fast=false          
2022-09-09 17:24:39.680729: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-09 17:24:39.686614: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-09 17:24:39.687162: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.6.0 at http://localhost:6006/ (Press CTRL+C to quit)
W0909 17:27:00.097848 140448668829440 security_validator.py:46] In 3.0, this warning will become an error:
Illegal Content-Security-Policy for script-src: 'unsafe-inline'
W0909 17:28:04.221373 140448668829440 security_validator.py:46] In 3.0, this warning will become an error:
Illegal Content-Security-Policy for script-src: 'unsafe-inline'
2022-09-09 17:28:37.258148: W tensorflow/core/profiler/convert/xplane_to_tools_data.cc:191] Could not find host XPlane for tf data stats: 
W0909 17:31:33.656232 140448668829440 security_validator.py:46] In 3.0, this warning will become an error:
Illegal Content-Security-Policy for script-src: 'unsafe-inline'
2022-09-09 17:31:40.786053: W tensorflow/core/profiler/convert/xplane_to_tools_data.cc:191] Could not find host XPlane for tf data stats: