Classify graph-replay synthetic ops by GPU kernel name#677
Classify graph-replay synthetic ops by GPU kernel name#677rkarhila-amd wants to merge 14 commits into
Conversation
|
The regression test fails because in earlier version any operations inside cuda graph were categorised as "other" whereas this PR gives them explicit categorisation.
This PR cannot pass the test as it makes a more fine grained analysis. We need this feature implemented in this way or another. Codeowners please comment @ajassani @gabeweisz @AMD-AGI/perf-opt |
The thing to do in this case is to update the reference outputs. We will be able to see how they changed in the deltas in the CSVs and make sure that they make sense. I think we have a script that you can use to update the reference results. If not, you can manually find the right files or ask Cursor or whichever tool you like to do it for you |
| ) | ||
| return None | ||
| ktl, knl = kt.lower(), kernel_name.lower() | ||
| if kt == "KV Cache Store": |
There was a problem hiding this comment.
I'd prefer to see a loop over a dict rather than this long list of if statements - I think that is a bit more maintainable
There was a problem hiding this comment.
Agreed, mappings should be in same format as for the non-graph kernels.
While working on this, I understood that the suggested change only affects reporting, and perf models are not applied to the kernels in graphs.
I'll see how massive overhaul it takes to get that bit working.
1955ada to
b2a44a8
Compare
|
Turned out to be quite an overhaul. Currently developed with only a single trace. I'll test with a few more cases tomorrow. |
|
The PR affects almost 80 files but majority of them are test cases. PR covers now the kernel types inside graphs that were listed in PR description chart. Still tested only on sglang, |
25319cc to
93acdaf
Compare
|
This looks like its working satisfactorily now. I'm running a final end-to-end test on sglang side. Still missing vllm test case, I'll get it tomorrow. FYI @devalshahamd I had an improperly activated patch in last traces, now I'm testing with properly instrumented Kimi-K2.6-MXFP4 traces accessible to AMDeers in sharepoint. |
b68d10e to
21d9435
Compare
|
Finally verified that these changes didn't break vllm trace handling. Please review now. If this is too much in one go, I can try to break it up to parts, for example I put considerable work on (directing cursor to) optimize the perf tree building in trace merging, and reduced the compute time for some of the operations from hours to seconds. (No matter how I try to avoid it, I somehow always find my way back to tweaking transducers and graph operations). That could be its own PR. The rest is difficult to split. |
0c3c6dc to
ba7846a
Compare
…etrics - Classify hipGraphLaunch/cudaGraphLaunch synthetic ops by underlying GPU kernel name - Resolve perf models for synthetics via SGLang capture linking and kernel heuristics - Compute roofline metrics for off-tree synthetics via gpu_events - Use per-kernel duration instead of hipGraphLaunch envelope time for synthetics Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Add capture-link augmentation for truncated MoE shapes and pointer-only RMSNorm launches, register dedicated perf models for kernel_moe_gemm and dynamic_per_group_scaled_quant, and prefer those rules over generic kernel-type mappings so graph-replay synthetics get roofline metrics in analysis. Co-authored-by: Cursor <cursoragent@cursor.com>
Register MoE sorting/top-k and decode-attention perf models, fix RMSNorm ScalarList dtype fallback, and expose roofline_time_fraction in priority_data. Co-authored-by: Cursor <cursoragent@cursor.com>
…tion. Update checked-in CSV golden files after graph-replay kernels move from other to SDPA_fwd, elementwise, and MoE_aux and gain perf models. Co-authored-by: Cursor <cursoragent@cursor.com>
Match surrounding KERNEL_TYPE_RULES formatting in classify_kernels.py. Co-authored-by: Cursor <cursoragent@cursor.com>
Reformat classify_kernels, perf_model_extensions, and trace_sglang_capture_link so lint CI passes with the pinned Black version. Co-authored-by: Cursor <cursoragent@cursor.com>
Graph launch nodes get execute_* annotations; synthetic GPU ops inherit annotations from ancestors. graph_decode_attention_kernel reads parsed sq/sk for N_Q/N_KV. UserWarnings flag defaulted dims/dtypes in graph replay perf models. Tests cover annotation parsing and SDPA params. Co-authored-by: Cursor <cursoragent@cursor.com>
Re-run formatter against merge-base with origin/main so CI lint stays green. Co-authored-by: Cursor <cursoragent@cursor.com>
Add time-sorted event indexing for capture merge, tqdm milestones during TreePerfAnalyzer init and pseudo-op extensions, parent-pointer cycle guards, and targeted pseudo-op kernel detection to avoid full-tree scans on large traces. Co-authored-by: Cursor <cursoragent@cursor.com>
MoE fused uses a one-pass subtree flag for cpu_op descendants; unfused Triton batches parent child-list rewrites. MLA decode/prefill use a stage1/donor index plus batched cpu_root_nodes updates via inject_pseudo_op_wrap_children (cpu_roots_acc) to avoid O(M×R) list rescans. apply_pseudo_op_extensions logs MoE branch choice, extension order, per-extension timings, and supports TRACELENS_PSEUDO_OPS_LOG for stderr diagnostics without --show-progress. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Serialize grouped call stacks as strings, strip parent_module whitespace, and drop per-kernel call_stack from unified summaries so graph inference golden CSVs match again. Co-authored-by: Cursor <cursoragent@cursor.com>
Update sglang_decode and sglang_prefilldecode reference CSVs to reflect call_stack enrichment and synthetic-op categorization from this branch; the golden CSVs added to main in #791 pre-date these features. Apply Black 26.1.0 to tree_perf.py (touched by the rebase onto main). Co-authored-by: Cursor <cursoragent@cursor.com>
ba7846a to
bd6c955
Compare
2 participants
Summary
categorize_torch_op()so TreePerf placeholder ops ({parent}->{kernel} (Synthetic Op)) are categorized from the underlying GPU kernel viaclassify_kernels, not the graph-launch parent name.GEMM,MoE_unfused,GroupQuant,RMSNorm,elementwise,MoE_aux, etc.) instead ofother.tests/test_synthetic_op_kernel_categorization.py.Sample output
Analysis of SGLang Minimax 2.5 traces. Before this fix, everything launched in graphs was categorised as "other" and no rooflines are provided. After the fix, kernels in graphs are categorised correctly and rooflines and potential improvements are calculated.
Top Operations without fix on main branch 2026/06/08:
Top Operations with this fix
Smoketest on vllm
Top operations from analysis of a small Qwen3-0.6B vllm trace with graphs. Kernels in graphs are found and rooflines are estimated.
Context
Graph-replay traces often attribute GPU work under
hipGraphLaunch/cudaGraphLaunchwithout a normalaiter::/aten::CPU op. This is the open-source counterpart to TraceLens-internal #458.Regression test updates
Refreshed checked-in CSV golden files (commit
23e80f9a) after intentional reporting changes from synthetic-op categorization and new graph-replay perf models. All 43 targeted regression cases now pass locally.test_perf_report_regression.pyperf_model/normalization)Parent op category:other→SDPA_fwd/elementwise;has_perf_model:False→True; normalization trace row order + RMSNormperf_params(ScalarList dtype fallback)test_compare_perf_reports.pycompare_test_e2eh100 + mi300 Qwen referencesothertest_inference_perf_report.pygraph_full,graph_piecewiseother;has_perf_modelflags settest_detect_recompute.pymi300/resnet_act_checkpointunified_perf_summaryrow order shifts when grouping keys changeTrace directories touched (62 CSV files):
h100/*(6 workloads),mi300/*(5 workloads + recompute),compare_test_e2e/reference/*,inference/graph_full,inference/graph_piecewise,perf_model/normalization.Unchanged (still pass):
mi210/gpu_only_trace,torch_compile_triton/trace.Test plan
pytest tests/test_synthetic_op_kernel_categorization.py -vpytest tests/test_perf_report_regression.py tests/test_compare_perf_reports.py tests/test_inference_perf_report.py tests/test_detect_recompute.pyotherbucket shrinks andgemm_ops.csv/moe_unfused_ops.csvpopulateMade with Cursor