Tasks are always in a delayed execution state and cannot be executed properly

[fangchongde]

The task has been executing in a delayed state, and the master's log is as follows:
[INFO] 2023-08-04 16:04:13.094 +0800 org.apache.dolphinscheduler.server.master.processor.TaskRecallProcessor:[61] - [WorkflowInstance-3417][TaskInstance-3465] - Receive task recall command: TaskRejectCommand(super=BaseCommand(messageSenderAddress=10.11.0.13:1234, messageReceiverAddress=10.11.0.6:5678, messageSendTime=1691136247681), taskInstanceId=3465, host=10.11.0.13:1234, processInstanceId=3417)
[INFO] 2023-08-04 16:04:13.364 +0800 org.apache.dolphinscheduler.server.master.processor.queue.TaskExecuteRunnable:[56] - [WorkflowInstance-3417][TaskInstance-3465] - Handle task event begin: TaskEvent(taskInstanceId=3465, workerAddress=null, state=null, startTime=null, endTime=null, executePath=null, logPath=null, processId=0, appIds=null, event=WORKER_REJECT, varPool=null, channel=[id: 0x0f8e8c97, L:/10.11.0.6:5678 - R:/10.11.0.13:56990], processInstanceId=3417)
[INFO] 2023-08-04 16:04:13.366 +0800 TaskLogLogger-class org.apache.dolphinscheduler.server.master.runner.task.CommonTaskProcessor:[105] - [WorkflowInstance-3417][TaskInstance-3465] - Task 接头质控-单样本计算 is already running or delayed, no need to submit to task queue, taskState: TaskExecutionStatus{code=12, desc='delay execution'}
[INFO] 2023-08-04 16:04:13.388 +0800 org.apache.dolphinscheduler.server.master.processor.queue.TaskExecuteRunnable:[59] - [WorkflowInstance-3417][TaskInstance-3465] - Handle task event finished: TaskEvent(taskInstanceId=3465, workerAddress=null, state=null, startTime=null, endTime=null, executePath=null, logPath=null, processId=0, appIds=null, event=WORKER_REJECT, varPool=null, channel=[id: 0x0f8e8c97, L:/10.11.0.6:5678 - R:/10.11.0.13:56990], processInstanceId=3417)
[INFO] 2023-08-04 16:04:13.389 +0800 org.apache.dolphinscheduler.server.master.processor.queue.TaskExecuteThreadPool:[123] - [WorkflowInstance-0][TaskInstance-0] - [WorkflowInstance-3417] persist events succeeded
[INFO] 2023-08-04 16:08:03.095 +0800 org.apache.dolphinscheduler.server.master.service.MasterFailoverService:[146] - [WorkflowInstance-0][TaskInstance-0] - Master[10.11.0.6:5678] failover starting there are 2 workflowInstance may need to failover, will do a deep check, workflowInstanceIds: [3370, 3417]
[INFO] 2023-08-04 16:08:03.095 +0800 org.apache.dolphinscheduler.server.master.service.MasterFailoverService:[155] - [WorkflowInstance-3417][TaskInstance-0] - WorkflowInstance failover starting
[INFO] 2023-08-04 16:08:03.095 +0800 org.apache.dolphinscheduler.server.master.service.MasterFailoverService:[157] - [WorkflowInstance-3417][TaskInstance-0] - WorkflowInstance doesn't need to failover

[Cocytus316]

This is most likely due to a clock inconsistency between the master and worker servers