[Debugger V2] Tensorboard Not Showing All Graph Executions in div List

[bberlo]

Environment information (required)

For browser-related issues, please additionally specify:

• Browser type and version (e.g., Chrome 64.0.3282.140): Chrome 91.0.4472.114 (64-bit)
• Screenshot, if it’s a visual issue:

Issue description

Please describe the bug as clearly as possible.

When running tensorboard with a logdir that contains files generated via the tf.debugging.experimental.enable_dump_debug_info() function, during both training via the keras model.fit() function and inference via the keras model.evaluate() function, not all graph executions are reported in the div list in the upper right corner (see image).

Is there a way to force tensorboard to show all graph executions?
In case not, is there a way to force tensorboard to show a specific range of graph executions?

How can we reproduce the problem without additional resources (including external data files and proprietary Python modules)?

Possibly by running a large training and inference session which results in more than 883010 graph executions and debugging via the following function declaration:

tf.debugging.experimental.enable_dump_debug_info(
    dump_root=DUMP_ROOT,
    tensor_debug_mode='FULL_HEALTH',
    circular_buffer_size=-1
)

[psybuzz]

Thanks for the report. @caisq, do you know if there are known limitations such as a max # of graph executions?

[bberlo]

Dear all,

I have performed a github search with the following search query: 'repo:tensorflow/tensorboard path:/tensorboard/plugins/debugger_v2 1000'.
The number 1000 refers to the number of alerts which can be noted in the image, which, when hitting 1000 never increased afterwards.
I found something interesting in /tensorboard/tensorboard/plugins/debugger_v2/debug_data_multiplexer.py (lines 30-34):

# Default number of alerts per monitor type.
# Limiting the number of alerts is based on the consideration that usually
# only the first few alerting events are the most critical and the subsequent
# ones are either repetitions of the earlier ones or caused by the earlier ones.
DEFAULT_PER_TYPE_ALERT_LIMIT = 1000

Since its quite late where I am living now, I will check tomorrow if changing this variable will absolve the issue that I am currently experiencing.

Update 01/07/2021

I have git cloned the tensorboard repo into a lubuntu 64-bit VM, changed DEFAULT_PER_TYPE_ALERT_LIMIT to 2000000, and build via bazel.
What I have noticed is that the number of alerts does not cap at 1000, however, the div list still does not load more than 883010 graph executions.
Therefore, there must also be a max preventing more div list elements from loading.