-
Notifications
You must be signed in to change notification settings - Fork 5k
[Improvement-17724][TaskPlugin] EmrTask resource leak repair #17719
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should create an issue first and link to it. DSIP-23 is a main issue, and you need to create a sub issue. @niumy0701
done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR fixes resource leaks in the EMR task plugin by ensuring that the AWS EMR client is properly shut down after task execution completes or is cancelled. Previously, EMR client connections were not being released, which could lead to resource exhaustion over time in long-running DolphinScheduler instances.
- Added
emrClient.shutdown()calls intrackApplicationStatus()andcancelApplication()methods for both EmrJobFlowTask and EmrAddStepsTask - Refactored
EmrAddStepsTask.cancelApplication()with try-finally block to guarantee cleanup even when exceptions occur - Added test coverage to verify shutdown behavior in both success and failure scenarios
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java | Added emrClient.shutdown() calls in trackApplicationStatus() and cancelApplication() to fix resource leak |
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java | Added emrClient.shutdown() in trackApplicationStatus() and wrapped cancelApplication() in try-finally to ensure cleanup |
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTaskTest.java | Added test to verify shutdown is called when cancelling job flow |
| dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java | Added tests to verify shutdown is called in error scenarios when cancelling steps |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Outdated
Show resolved
Hide resolved
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Outdated
Show resolved
Hide resolved
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Show resolved
Hide resolved
...-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same #17718 (review)
There should be no problem, the resources have already been closed in the finally block |
Please refer to the details of main issue #14877
fix #17724
Purpose of the pull request
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(or)
Pull Request Notice
Pull Request Notice
If your pull request contains incompatible change, you should also add it to
docs/docs/en/guide/upgrade/incompatible.md