Skip to content

Commit

Permalink
[MINOR][CORE] When failed to canceling the job group, add a warning log
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
The pr aims to add add a warning log when failed to canceling the job group.

### Why are the changes needed?
In production, we found that when there are many jobs, if we specify `a just-submitted job group` and `cancel` it using the API `SparkContext#cancelJobGroup(groupId: String)`, the cancellation will fail (`silently`) because the job has not yet entered `activeJobs`, and there is `no warning` prompt.
(PS: Due to historical reasons, we are not currently using the newer API `SparkContext#cancelJobGroupAndFutureJobs(groupId: String)`)

I propose to add `a warning log` here to help troubleshoot `issues`.

### Does this PR introduce _any_ user-facing change?
Yes, only for log.

### How was this patch tested?
N/A

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#45722 from panbingkun/handleJobGroupCancelled_add_log.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
panbingkun authored and dongjoon-hyun committed Mar 26, 2024
1 parent fd4b8e8 commit 7d87a94
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1220,6 +1220,9 @@ private[spark] class DAGScheduler(
_.getProperty(SparkContext.SPARK_JOB_GROUP_ID) == groupId
}
}
if (activeInGroup.isEmpty && !cancelFutureJobs) {
logWarning(s"Failed to cancel job group $groupId. Cannot find active jobs for it.")
}
val jobIds = activeInGroup.map(_.jobId)
jobIds.foreach(handleJobCancellation(_,
Option("part of cancelled job group %s".format(groupId))))
Expand Down

0 comments on commit 7d87a94

Please sign in to comment.