[GPU]Performance optimizing to RMS and MVN#36747
Open
ceciliapeng2011 wants to merge 1 commit into
Open
Conversation
…MVN dispatching & kernel algorithm, and improving HW utilization to close to 100% bandwidth, clears 2 Omni targets.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Intel GPU kernel selector + OpenCL kernels for MVN and RMS “bfyx_opt” implementations to use a more generalized workgroup (LWS) selection strategy and adaptive register-caching vs reread behavior, aiming to improve throughput across varying sequence widths and dynamic shapes.
Changes:
- Generalized LWS selection (power-of-two, ~8 elements per work-item) for MVN and RMS selectors.
- Added JIT-controlled caching vs reread behavior (
*_STACK_SIZE,*_REREAD_INPUT) and new kernel code paths (vectorized IO for RMS; fused mean/variance accumulation for MVN). - Added RMS subgroup-aware reduction modes (
ONE_SUBGROUP_ROW/MULTI_SUBGROUP_ROW) and revised SLM usage.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/plugins/intel_gpu/src/kernel_selector/kernels/rms/rms_kernel_bfyx_opt.cpp | Generalized RMS dispatch/JIT constants (LWS selection, stack sizing, reread flags, subgroup-row mode flags). |
| src/plugins/intel_gpu/src/kernel_selector/kernels/mvn/mvn_kernel_bfyx_opt.cpp | Generalized MVN dispatch/JIT constants (LWS selection, stack sizing, reread flags, dynamic-vs-static JIT control). |
| src/plugins/intel_gpu/src/kernel_selector/cl_kernels/rms_gpu_bfyx_opt.cl | RMS kernel refactor: vectorized reads/writes, subgroup-row reduction variants, cache-or-reread input path. |
| src/plugins/intel_gpu/src/kernel_selector/cl_kernels/mvn_gpu_bfyx_opt.cl | MVN kernel refactor: single-pass sum+sumSq accumulation, cache-or-reread input path, updated normalization math. |
Comment on lines
+86
to
88
| float red_sq = work_group_reduce_add(my_sq); | ||
| float my_variance = red_sq / data_set_size - my_sum_mean * my_sum_mean; | ||
| # if defined EPS_OUTSIDE_SQRT |
Comment on lines
+117
to
+120
| if (is_decimal_number(data_set_size)) { | ||
| const size_t static_data_set_size = std::stoul(data_set_size); | ||
| const size_t lws = get_generalized_lws(static_data_set_size, dispatchData.maxSlmSize); | ||
| const size_t required_stack = get_stack_size(static_data_set_size, lws); |
Comment on lines
123
to
+127
| MakeJitConstant("DATA_SIZE", data_size), | ||
| MakeJitConstant("LWS", lws_0), | ||
| MakeJitConstant("SLM_SIZE", dispatchData.maxSlmSize), | ||
| MakeJitConstant("STACK_SIZE", stack_size) | ||
| MakeJitConstant("STACK_SIZE", stack_size), | ||
| MakeJitConstant("SUBGROUP_BLOCK_SIZE", 8), |
e-ddykim
approved these changes
Jul 7, 2026
e-ddykim
left a comment
e-ddykim
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me. Please check my comments.
| uses_runtime_lws = false; | ||
| } | ||
| jit.AddConstants({ | ||
| MakeJitConstant("IS_DYNAMIC", uses_runtime_lws), |
Contributor
There was a problem hiding this comment.
IS_DYNAMIC is already defined here.
This could trigger a macro redefinition error in a debug build. If overriding the existing value is intended, please consider removing the definition from kernel_base.
Comment on lines
45
to
51
| #if SUBGROUP_BLOCK_SIZE == 1 | ||
| #define BLOCK_READ(ptr, offset) DT_INPUT_BLOCK_READ(ptr, offset) | ||
| #define BLOCK_WRITE(ptr, offset, val) DT_OUTPUT_BLOCK_WRITE(ptr, offset, val) | ||
| #define ACC_TYPE ACCUMULATOR_TYPE | ||
| #define TO_ACC_TYPE(x) TO_ACCUMULATOR_TYPE(x) | ||
| #define OUTPUT_VEC_TYPE OUTPUT_TYPE | ||
| #else |
Contributor
There was a problem hiding this comment.
SUBGROUP_BLOCK_SIZE is always 8 now. Please check whether these dead codes can be safely removed.
p-durandin
requested changes
Jul 8, 2026
p-durandin
left a comment
p-durandin
left a comment
Contributor
There was a problem hiding this comment.
Looks like extension of #36587
@ceciliapeng2011 please address @alvoron comments here
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance optimize to mvn_gpu_bfyx_opt and rms_gpu_bfyx_opt kernels.
Details:
Implemented a generalized MVN scheduling rule that selects the largest power-of-two workgroup size while keeping about 8 normalized elements per work-item, replacing shape-specific tuning. Added adaptive register-caching with automatic fallback to reread mode when stack pressure is high, so the kernel keeps high bandwidth efficiency across varying sequence widths and dynamic shapes.
Implemented the same generalized workgroup selection strategy for RMS, with subgroup-aware reduction paths: one-subgroup rows bypass SLM/barriers, while multi-subgroup rows use lightweight partial reduction in SLM. Added adaptive cache-or-reread behavior plus vectorized IO fast paths, improving robustness and throughput for generalized (non-bucketed) shapes.
Tickets:
AI Assistance: