Experimental multi stream support for graph capture and graph replay trace merge merge#776
Draft
devalshahamd wants to merge 1 commit into
Draft
Experimental multi stream support for graph capture and graph replay trace merge merge#776devalshahamd wants to merge 1 commit into
devalshahamd wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request significantly improves the accuracy and reliability of pairing kernel launch events between capture and graph traces in multi-stream GPU traces. The main enhancement is the introduction of a stream-aware pairing algorithm that matches events based on both stream and kernel name, replacing the previous positional matching logic. This ensures correct alignment even when event orders differ across streams.
The most important changes are:
Stream-aware event pairing and verification:
_normalize_kernel_namefunction to consistently extract and normalize kernel names across different trace formats, collapsing memory operations to stable tokens for robust matching.build_stream_pairsfunction to pair graph kernels and capture launches by stream and name, handling multi-stream traces and ensuring each event is paired exactly once.verify_subtree_eventsto use the new stream-aware pairing, returning success only if all events are correctly paired.Refactoring and integration:
merge_capture_trace_into_graphto compute event pairs once usingbuild_stream_pairs, using the result for both verification and connection, reducing redundant computation and improving clarity. [1] [2]make_connectionsto accept precomputed event pairs instead of separate event lists, simplifying its interface and leveraging the new pairing logic. [1] [2]These changes make the kernel launch pairing process more robust, especially for traces with multiple GPU streams.