Skip to content

Conversation

@niumy0701
Copy link

@niumy0701 niumy0701 commented Nov 25, 2025

Please refer to the details of main issue #14877
fix #17724

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

Copy link
Member

@SbloodyS SbloodyS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should create an issue first and link to it. DSIP-23 is a main issue, and you need to create a sub issue. @niumy0701

@niumy0701
Copy link
Author

You should create an issue first and link to it. DSIP-23 is a main issue, and you need to create a sub issue. @niumy0701

done
sub issue is #17724

@niumy0701 niumy0701 changed the title [DSIP-23][TaskPlugin] EmrTask resource leak repair [Improvement-17724][TaskPlugin] EmrTask resource leak repair Nov 25, 2025
@niumy0701 niumy0701 requested a review from SbloodyS November 25, 2025 08:02
@SbloodyS SbloodyS requested a review from Copilot November 26, 2025 02:00
Copilot finished reviewing on behalf of SbloodyS November 26, 2025 02:03
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes resource leaks in the EMR task plugin by ensuring that the AWS EMR client is properly shut down after task execution completes or is cancelled. Previously, EMR client connections were not being released, which could lead to resource exhaustion over time in long-running DolphinScheduler instances.

  • Added emrClient.shutdown() calls in trackApplicationStatus() and cancelApplication() methods for both EmrJobFlowTask and EmrAddStepsTask
  • Refactored EmrAddStepsTask.cancelApplication() with try-finally block to guarantee cleanup even when exceptions occur
  • Added test coverage to verify shutdown behavior in both success and failure scenarios

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java Added emrClient.shutdown() calls in trackApplicationStatus() and cancelApplication() to fix resource leak
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java Added emrClient.shutdown() in trackApplicationStatus() and wrapped cancelApplication() in try-finally to ensure cleanup
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTaskTest.java Added test to verify shutdown is called when cancelling job flow
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java Added tests to verify shutdown is called in error scenarios when cancelling steps

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@niumy0701
Copy link
Author

same #17718 (review)

There should be no problem, the resources have already been closed in the finally block

@niumy0701 niumy0701 requested a review from ruanwenjun November 29, 2025 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement][TaskPlugin] EmrTask resource leak issue

3 participants