Skip to content

Experimental multi stream support for graph capture and graph replay trace merge merge#776

Draft
devalshahamd wants to merge 1 commit into
mainfrom
feat/graph_capture_merge_multistream
Draft

Experimental multi stream support for graph capture and graph replay trace merge merge#776
devalshahamd wants to merge 1 commit into
mainfrom
feat/graph_capture_merge_multistream

Conversation

@devalshahamd

Copy link
Copy Markdown
Contributor

This pull request significantly improves the accuracy and reliability of pairing kernel launch events between capture and graph traces in multi-stream GPU traces. The main enhancement is the introduction of a stream-aware pairing algorithm that matches events based on both stream and kernel name, replacing the previous positional matching logic. This ensures correct alignment even when event orders differ across streams.

The most important changes are:

Stream-aware event pairing and verification:

  • Added the _normalize_kernel_name function to consistently extract and normalize kernel names across different trace formats, collapsing memory operations to stable tokens for robust matching.
  • Introduced the build_stream_pairs function to pair graph kernels and capture launches by stream and name, handling multi-stream traces and ensuring each event is paired exactly once.
  • Updated verify_subtree_events to use the new stream-aware pairing, returning success only if all events are correctly paired.

Refactoring and integration:

  • Modified merge_capture_trace_into_graph to compute event pairs once using build_stream_pairs, using the result for both verification and connection, reducing redundant computation and improving clarity. [1] [2]
  • Refactored make_connections to accept precomputed event pairs instead of separate event lists, simplifying its interface and leveraging the new pairing logic. [1] [2]

These changes make the kernel launch pairing process more robust, especially for traces with multiple GPU streams.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants