-
Notifications
You must be signed in to change notification settings - Fork 744
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GOBBLIN-2143] handle concurrent ReevaluateDagProc for cancelled dag nodes correctly #4038
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -134,7 +134,7 @@ public static void submitJobToExecutor(DagManagementStateStore dagManagementStat | |
jobOrchestrationTimer.stop(jobMetadata); | ||
log.info("Orchestrated job: {} on Executor: {}", DagManagerUtils.getFullyQualifiedJobName(dagNode), specExecutorUri); | ||
dagManagementStateStore.getDagManagerMetrics().incrementJobsSentToExecutor(dagNode); | ||
dagManagementStateStore.addDagNodeState(dagNode, dagId); | ||
dagManagementStateStore.updateDagNode(dagNode); | ||
sendEnforceJobStartDeadlineDagAction(dagManagementStateStore, dagNode); | ||
} catch (Exception e) { | ||
TimingEvent jobFailedTimer = DagProc.eventSubmitter.getTimingEvent(TimingEvent.LauncherTimings.JOB_FAILED); | ||
|
@@ -158,7 +158,6 @@ public static void submitJobToExecutor(DagManagementStateStore dagManagementStat | |
|
||
public static void cancelDagNode(Dag.DagNode<JobExecutionPlan> dagNodeToCancel, DagManagementStateStore dagManagementStateStore) throws IOException { | ||
Properties cancelJobArgs = new Properties(); | ||
DagManager.DagId dagId = DagManagerUtils.generateDagId(dagNodeToCancel); | ||
String serializedFuture = null; | ||
|
||
if (dagNodeToCancel.getValue().getJobSpec().getConfig().hasPath(ConfigurationKeys.FLOW_EXECUTION_ID_KEY)) { | ||
|
@@ -175,12 +174,6 @@ public static void cancelDagNode(Dag.DagNode<JobExecutionPlan> dagNodeToCancel, | |
log.warn("No Job future when canceling DAG node - {}", dagNodeToCancel.getValue().getId()); | ||
} | ||
DagManagerUtils.getSpecProducer(dagNodeToCancel).cancelJob(dagNodeToCancel.getValue().getJobSpec().getUri(), cancelJobArgs).get(); | ||
// add back the dag node with updated states in the store | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. addDag will happen only by LaunchDagProc once in the life cycle of Dag There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. updating the status too soon (like here), may cause a dag to delete before ReevaluateDagProc is processed. |
||
dagNodeToCancel.getValue().setExecutionStatus(ExecutionStatus.CANCELLED); | ||
dagManagementStateStore.addDagNodeState(dagNodeToCancel, dagId); | ||
// send cancellation event after updating the state, because cancellation event triggers a ReevaluateDagAction | ||
// that will delete the dag. Due to race condition between adding dag node and deleting dag, state store may get | ||
// into inconsistent state. | ||
sendCancellationEvent(dagNodeToCancel); | ||
log.info("Cancelled dag node {}, spec_producer_future {}", dagNodeToCancel.getValue().getId(), serializedFuture); | ||
} catch (Exception e) { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
only
.info
level, notwarn
or evenerror
? alternatively this could arguably be an exceptionThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i mean, deletion not happening can really be just because the element was already absent...
if there is some mysql exception, it would be thrown, it will not return a boolean in that case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be try/catch in this case with logging?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i mean, i intend to not swallow exceptions.
and i am just fine if cleanup returns false. so did not do try/catch