Introduce PerProcessContext to adjust DDP per process behavior (#913) #955
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Therefore, this PR targets the combined problem of #953 and #954
Solution
Introduce a global singleton called
PerProcessContext
. As its name suggests, this context maintains properties that are private to each process. In the current implementation,PerProcessContext
allows accessing the rank (process ID) and the total number of processes from anywhere.Design Choices:
run_under_record_context
as a whole instead of just hijacking_cond
. This is becauserun_under_record_context
will initialize the writer and the write the global steps before the first call to_cond
, and we would like to avoid that as well.Testing