BeforeHookCreation Hook Deletion Policy not working

[TalhaNaeem101]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
I have configured argocd hooks on job with following configurations, but the hook does not get the job deleted before the creation of new job after some update unlike mentioned over here in the doc. And which eventually causes the appliation to hang in progressing state and the job also not get recreated with new updates.
Ref: https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/#hook-deletion-policies

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
  generation: 2
  labels:
    app: asset-service-postgres-migration-debugging
  name: asset-service-postgres-migration-debugging
  namespace: dev
spec:
  backoffLimit: 1
  completionMode: NonIndexed
  completions: 1
  parallelism: 1
  .
  .
  .
  ttlSecondsAfterFinished: 140

To Reproduce
Create a simple helm chart and put the following job in it and keep the helm chart within any of your github repos.

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
  labels:
    app: asset-service-dummy-migration-job
    app.kubernetes.io/instance: assets-service-migration-debugging
  name: asset-service-dummy-migration-job
  namespace: dev
  resourceVersion: '332550705'
  uid: cfa739b4-ce4f-47b8-a921-72a9b8a269aa
spec:
  backoffLimit: 1
  completionMode: NonIndexed
  completions: 1
  parallelism: 1
  template:
    metadata:
      labels:
        app: asset-service-dummy-migration-job
    spec:
      containers:
        - command:
            - echo
            - "helo"         
          image:  busybox
          imagePullPolicy: Always
          name: asset-service-dummy-migration-job
      initContainers:
        - command:
            - /bin/sh
            - '-c'
            - |
              echo "ehlo"
          image: 'busybox:latest'
          name: init-ds
      restartPolicy: Never
  ttlSecondsAfterFinished: 140

Prepare application set for deploying the helm chart, update the following manifest:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: talha-migration-debugging
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: <YOUR GITHUB REPOSITORY URL>
      revision: <YOUR BRANCH NAME>
      directories:
      - path: <PATH OF THE HELM CHART WITHIN YOUR REPOSITORY>
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      syncPolicy:
        syncOptions:
        - CreateNamespace=true
        automated: 
          prune: false
          selfHeal: false
      source:
        repoURL: <YOUR GITHUB REPOSITORY URL>
        targetRevision: <YOUR BRANCH NAME>
        path: '{{path}}'
        helm:
          valueFiles:
           - values-talha.yaml
      destination:
        server: https://kubernetes.default.svc
        namespace: dev

Create the application set:

kubectl apply -f applicationset.yaml

The job should be created and run successfully. Next, you need to add some trigger in the job manifest and see the job (as hook) will not update itself automatically. But it will stuck in progressing state. And it will show the reason of the job sync, Pending Deletion.

Expected behavior

Screenshots

Version

argocd: v2.7.8+92949f6.dirty
  BuildDate: 2023-07-19T15:18:54Z
  GitCommit: 92949f6033080f2fba9a9654f46592fb71982770
  GitTreeState: dirty
  GoVersion: go1.19.10
  Compiler: gc
  Platform: linux/amd64

Logs

time="2023-08-07T11:30:36Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.content="name:\"assets-service-migration-debugging\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-08-07T11:30:36Z" span.kind=server system=grpc

[ShadaabSikandar]

Facing the same issue

[djhoese]

Just curious, does your second trigger/sync happen before the Job is deleted due to "ttlSecondsAfterFinished"? Or after (meaning the Job has already been deleted)?

[djhoese]

Doing some searches shows:

argoproj/gitops-engine#461

[crenshaw-dev]

If someone could open an Argo CD PR pointing at the gitops-engine PR to show that tests pass, that'll help us move the gitops-engine PR forward.

[TalhaNaeem101]

due to "ttlSecondsAfterFinished"? Or after (meaning the Job has already been deleted)?

@djhoese Both ways, it gets the application stuck in progressing state. Meaning that 1) With ttlSecondsAfterFinished set to 4-5 mins-- and 2) ttlSecondsAfterFinished not set.
The hook does not work ideally. And yes the issue might be the same as you mentioned, the job does not get deleted, and when we sync the application, the sync does not succeed as the job is immutable and hence it can not update it.

[sonamkshenoy]

I tried to reproduce the bug with the provided configurations of a BeforeHookCreation deletion policy and a Pre-sync hook, as well as a few other combinations. However, the BeforeHookCreation policy worked as expected in all the cases, and I was unable reproduce the error..

[talha-naeem1]

awesome, but Which argocd version do you have? @sonamkshenoy

[cehoffman]

We are seeing this too and seem to have correlated down to the inclusion of the ServerSideApply=true option on sync. Removing that option results in the Job being deleted. This is with 2.9.2 ArgoCD, but I'm fairly certain we've had this issue for a while, at least going back to 2.8 series.