[CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods #2985
Conversation
|
Cool. Do test this PR on the cluster, if something is missing, it will cause data correctness problems which are important. |
Well, i'll test normal and abnormal cases in yarn cluster laster. |
|
@zwangsheng Hi, How are your tests? |
Tested on yarn cluster, passing normal Terasort tests, as well as abnormal random task failure tests. But for the sake of correctness, I still need time to continue testing. |
|
Test the normal case(with terasort test) and the abnormal case(RDD with task fail), seems this change is safe. @FMX plz take a review |
OK, please solve the conflicts. The review will be done soon. |
zwangsheng
force-pushed
the
CELEBORN-1768
branch
from
73730f6 to
52da4bd
Compare
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This issue was closed because it has been staled for 10 days with no activity. |
|
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
What changes were proposed in this pull request?
Refactoring shuffleWriters, extract common methods as BasedShuffleWriter.
Why are the changes needed?
Currently, HashBasedShuffleWriter and SortBasedShuffleWriter have a lot of the same methods, which makes it difficult for us to maintain the code. In many scenarios, we need to update multiple classes at the same time.
Therefore, we need to refactor Shuffle Writer, extract common methods without changing the specific method logic, and simplify the corresponding Shuffle Writer.
Does this PR introduce any user-facing change?
No, we do not change the logic code of apis or others.
How was this patch tested?
UT and local env