Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Introduce the disk buffer in WriteBufferManager for low memory executor in spark uniffle client #2344

Open
2 of 3 tasks
zuston opened this issue Jan 16, 2025 · 2 comments
Open
2 of 3 tasks

[FEATURE] Introduce the disk buffer in WriteBufferManager for low memory executor in spark uniffle client #2344

zuston opened this issue Jan 16, 2025 · 2 comments

Comments

@zuston
Copy link
Member

zuston commented Jan 16, 2025

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the feature

I found some jobs failed due to the failure of memory requiring in low memory spark task executor. For this case, we may introduce the disk buffer to slow down the memory requirement.

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zuston
Copy link
Member Author

zuston commented Jan 16, 2025

cc @jerqi @maobaolong @xianjingfeng

@jerqi
Copy link
Contributor

jerqi commented Jan 17, 2025 •
edited
Loading

cc @jerqi @maobaolong @xianjingfeng

It seems ok for me if this optimization is optional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jerqi @zuston