Fix FlopsProfiler crash when dp_world_size is None under sequence parallelism#8122
Open
vineethsaivs wants to merge 1 commit into
Open
Conversation
…allelism
When Ulysses sequence parallelism is enabled the DeepSpeed engine reports
dp_world_size as None, since the data-parallel replication is folded into the
sequence-data-parallel group. print_model_profile formatted that value with
'{:<8}', which raised 'unsupported format string passed to NoneType.__format__'
and crashed the profiler at every step. Report the sequence-data-parallel world
size when dp_world_size is None.
Fixes deepspeedai#7483
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Ulysses sequence parallelism is enabled, the DeepSpeed engine reports
dp_world_sizeasNone(groups._get_data_parallel_world_size()returnsNonefor a sequence-parallelmpu, because the data-parallel replication is folded into the sequence-data-parallel group).FlopsProfiler.print_model_profile()formats that value with'{:<8}', and'{:<8}'.format(None)raises:so the profiler crashes on every profile step once sequence parallelism is on. This matches the traceback in #7483.
Fix
When
dp_world_sizeisNone, reportseq_dp_world_sizeinstead (the engine always sets it, and under sequence parallelism it is the effective data-parallel replication). The normal (non-sequence-parallel) path is unchanged.Test
Added
test_print_model_profile_with_none_dp_world_sizeintests/unit/profiling/flops_profiler/test_flops_profiler.py. It drivesprint_model_profilewith an engine whosedp_world_sizeisNone. It fails onmasterwith theNoneType.__format__TypeErrorand passes with this change, asserting the sequence-data-parallel world size is shown for "data parallel size".Fixes #7483