[Bug] Reallocating ShuffleServer functionality during the write phase takes up a lot of Driver memory.

[yl09099]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

When the Executor obtains the ShuffleHandleInfo object from the Driver, if the number of writers and readers is too large (the number of partitions is too large), the ShuffleServer function is reassigned during the write failure phase. GRPC thread ShuffleHandleInfo converted to RPC message, causing a large memory footprint, resulting in Driver crash.

Affects Version(s)

0.10.0

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!