Behavior of creating Argo Application thru API (crossplane) vs. Kubectl Apply #20580

patpicos · 2024-10-29T18:26:26Z

Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I've included steps to reproduce the bug.
I've pasted the output of argocd version.

Describe the bug

We are using crossplane to deploy/register applications against ArgoCD instances. When deploying through the API, we find that it is more strict than doing a kubectl apply with an equivalent manifest.

To Reproduce

YAML submitted by kubectl

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-amf
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    name: in-cluster
    namespace: argocd
  project: nk-rdc-us-east-1-argocd-prj-dev
  source:
    # Note that the path does not exist yet since we are registering the app. the configs will be populated later
    path: us-east-1/nk-rdc-amf-cicd-demo/workloads/amf
    repoURL: https://xxxxxx/v3-configuration-dev/isv-configuration/nk-rdc-dev/nk-rdc-dev.git 
    targetRevision: main
  syncPolicy:
    managedNamespaceMetadata:
      labels:
        gitops.xxx.io/cluster-name: nk-rdc-amf-cicd-demo
        gitops.xxx.io/cnf-name: amf
        gitops.xxx.io/environment: dev
        gitops.xxx.io/isv-name: nk-rdc
        gitops.xxx.io/region: us-east-1
        gitops.xxx.io/service-tier: nonp
    syncOptions:
    - Validate=false
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    - RespectIgnoreDifferences=true
    - ApplyOutOfSyncOnly=true

Equivalent resource from crossplane, the application never gets accepted and returns an error and seen as status. The issue with that is crossplane now HAMMERS the argocd API trying to recreate it because it was not accepted.

apiVersion: applications.argocd.crossplane.io/v1alpha1
kind: Application
metadata:
  annotations:
    crossplane.io/composition-resource-name: nk-rdc-dev-ghshh-nk-rdc-us-east-1-nk-rdc-amf-cicd-demo-amf-cnf
    crossplane.io/external-create-failed: '2024-10-29T17:20:35Z'
    crossplane.io/external-create-pending: '2024-10-29T17:20:34Z'
    crossplane.io/external-name: nk-rdc-dev-ghshh-nk-rdc-us-east-1-nk-rdc-amf-cicd-demo-amf-cnf
  creationTimestamp: '2024-10-29T16:28:18Z'
  finalizers:
    - finalizer.managedresource.crossplane.io
  generateName: nk-rdc-dev-ghshh-
  generation: 2
  labels:
    argocd.gitops.xxx/application-type: cnf-workload
    crossplane.io/claim-name: nk-rdc-dev
    crossplane.io/claim-namespace: isv-onboarding-workload-registration
    crossplane.io/composite: nk-rdc-dev-ghshh
    gitops.xxx/environment: dev
    gitops.xxx/isv-name: nk-rdc
    gitops.xxx/service-tier: nonp
  name: nk-rdc-dev-ghshh-nk-rdc-us-east-1-nk-rdc-amf-cicd-demo-amf-cnf
  ownerReferences:
    - apiVersion: onboarding.gitops.k8s.xxx/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: XCnfOnboarding
      name: nk-rdc-dev-ghshh
      uid: a1a198d9-21cc-4517-942a-a61197ee45fa
  resourceVersion: '12873336'
  uid: b613a01b-a3c7-4807-b5dd-ba4656aa1a1c
spec:
  deletionPolicy: Orphan
  forProvider:
    destination:
      name: in-cluster
      namespace: argocd
    finalizers:
      - resources-finalizer.argocd.argoproj.io
    project: nk-rdc-us-east-1-argocd-prj-dev
    source:
      path: us-east-1/nk-rdc-amf-cicd-demo/workloads/amf
      repoURL: >-
        https://xxxxv3-configuration-dev/isv-configuration/nk-rdc-dev/nk-rdc-dev.git
      targetRevision: main
    syncPolicy:
      managedNamespaceMetadata:
        labels:
          gitops.xxx/cluster-name: nk-rdc-amf-cicd-demo
          gitops.xxx/cnf-name: amf
          gitops.xxx/environment: dev
          gitops.xxx/isv-name: nk-rdc
          gitops.xxx/region: us-east-1
          gitops.xxx/service-tier: nonp
      syncOptions:
        - Validate=false
        - CreateNamespace=true
        - PrunePropagationPolicy=foreground
        - PruneLast=true
        - RespectIgnoreDifferences=true
        - ApplyOutOfSyncOnly=true
  managementPolicies:
    - Create
    - Delete
    - LateInitialize
    - Observe
    - Update
  providerConfigRef:
    name: rmc-us-east-1-dev-argocd-provider-config
status:
  atProvider:
    health: {}
    summary: {}
    sync:
      comparedTo:
        destination: {}
        source:
          repoURL: ''
      status: ''
  conditions:
    - lastTransitionTime: '2024-10-29T16:28:18Z'
      reason: Creating
      status: 'False'
      type: Ready
    - lastTransitionTime: '2024-10-29T17:02:14Z'
      message: >-
        create failed: cannot create Argocd application: rpc error: code =
        InvalidArgument desc = application spec for
        nk-rdc-dev-ghshh-nk-rdc-us-east-1-nk-rdc-amf-cicd-demo-amf-cnf is
        invalid: InvalidSpecError: Unable to generate manifests in
        us-east-1/nk-rdc-amf-cicd-demo/workloads/amf: rpc error: code = Unknown
        desc = Manifest generation error (cached):
        us-east-1/nk-rdc-amf-cicd-demo/workloads/amf: app path does not exist
      reason: ReconcileError
      status: 'False'
      type: Synced

Expected behavior

When submitting the app through the API (ie thru crossplane), we expect the application to be accepted and end up in Unknown status while the contents in the folder get populated

Screenshots

Proof kubectl accepts the manifest

k get app test-amf -o yaml | yq .status                                                                                                                                                                                                                                                                                                        <aws:DishWPaaSOperator-MGMT-DEV>
conditions:
  - lastTransitionTime: "2024-10-29T18:18:48Z"
    message: 'Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): us-east-1/nk-rdc-amf-cicd-demo/workloads/amf: app path does not exist'
    type: ComparisonError
controllerNamespace: argocd
health:
  status: Healthy
reconciledAt: "2024-10-29T18:18:48Z"
sync:
  comparedTo:
    destination:
      name: in-cluster
      namespace: argocd
    source:
      path: us-east-1/nk-rdc-amf-cicd-demo/workloads/amf
      repoURL: https://xxxxx/v3-configuration-dev/isv-configuration/nk-rdc-dev/nk-rdc-dev.git
      targetRevision: main
  status: Unknown

Perspective on the Crossplane managed resource:

Version

Paste the output from `argocd version` here.
argocd: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T15:30:40Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.23.0
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T15:30:40Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.23.0
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.4.3 2024-07-19T16:34:19Z
  Helm Version: v3.16.1+g5a5449d
  Kubectl Version: v0.29.6
  Jsonnet Version: v0.20.0

Logs

Paste any relevant application logs here.

from ArgoCD server

time="2024-10-29T18:30:24Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for nk-rdc-dev-ghshh-nk-rdc-us-east-1-nk-rdc-amf-cicd-demo-amf-cnf is invalid: InvalidSpecError: Unable to generate manifests in us-east-1/nk-rdc-amf-cicd-demo/workloads/amf: rpc error: code = Unknown desc = Manifest generation error (cached): us-east-1/nk-rdc-amf-cicd-demo/workloads/amf: app path does not exist" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2024-10-29T18:30:24Z" grpc.time_ms=632.632 span.kind=server system=grpc

The text was updated successfully, but these errors were encountered:

patpicos · 2024-10-29T18:33:49Z

So, is there an option to make the API behave like kubectl apply? I guess a fallback could be to use the kubernetes provider in crossplane to submit the app instead of using the crossplane provider which interacts with the argocd-server..

blakepettersson · 2024-10-29T20:03:37Z

The issue with that is crossplane now HAMMERS the argocd API trying to recreate it because it was not accepted.

I'm not sure why this is an issue with Argo CD. IMO that sounds more like an issue with the Argo CD Provider.

patpicos · 2024-10-29T20:09:01Z

ill double check if it actually hammers or its just the UI making it look it is repeatedly calling the API (the resource keeps getting the square animation around it). If the provider doesnt exponentially requeue, I will log it on the provider.

In any case, I am more curious about the API vs. kubectl apply behavior for the App creation and if there is a way to have similar behavior....before i commit to using the kubernetes provider

patpicos · 2024-10-29T20:12:43Z

Update, it is indeed hammering (2-3x per second). I also see that the provider is issueing a list command which is causing a panic

time="2024-10-29T20:10:24Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:24Z" grpc.time_ms=1.054 span.kind=server system=grpc
time="2024-10-29T20:10:24Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1728593133,\"iss\":\"argocd\",\"jti\":\"1a61da56-7ebf-428e-8f2e-d4bea7206c92\",\"nbf\":1728593133,\"sub\":\"crossplane-provider\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:24Z" span.kind=server system=grpc

time="2024-10-29T20:10:25Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for mt-ndc-dev-jzsrn-mt-ndc-us-east-1-mt-ndc-chf-cicd-chf-cnf is invalid: InvalidSpecError: Unable to generate manifests in us-east-1/mt-ndc-chf-cicd/workloads/chf: rpc error: code = Unknown desc = Manifest generation error (cached): us-east-1/mt-ndc-chf-cicd/workloads/chf: app path does not exist" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:24Z" grpc.time_ms=627.098 span.kind=server system=grpc
time="2024-10-29T20:10:25Z" level=info msg="received unary call /application.ApplicationService/List" grpc.method=List grpc.request.claims="{\"iat\":1728593133,\"iss\":\"argocd\",\"jti\":\"1a61da56-7ebf-428e-8f2e-d4bea7206c92\",\"nbf\":1728593133,\"sub\":\"crossplane-provider\"}" grpc.request.content="name:\"nk-rdc-dev-ghshh-nk-rdc-us-east-1-nk-rdc-amf-cicd-demo-amf-cnf\" " grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:25Z" span.kind=server system=grpc

time="2024-10-29T20:10:25Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:25Z" grpc.time_ms=0.568 span.kind=server system=grpc
time="2024-10-29T20:10:25Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1728593133,\"iss\":\"argocd\",\"jti\":\"1a61da56-7ebf-428e-8f2e-d4bea7206c92\",\"nbf\":1728593133,\"sub\":\"crossplane-provider\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:25Z" span.kind=server system=grpc

r-nasiri · 2024-10-30T01:45:11Z

The issue with that is crossplane now HAMMERS the argocd API trying to recreate it because it was not accepted.

I'm not sure why this is an issue with Argo CD. IMO that sounds more like an issue with the Argo CD Provider.

if you use the cli command with a path that doesn't exist, it gives you the same error message. I think the expectation is to create the app but leave it in not synced state, similar to behaviour when you create the app using a manifest

argocd app create test-app --repo  https://github.com/argoproj/argocd-example-apps.git --path bad-path --dest-namespace default --dest-server https://kubernetes.default.svc 
FATA[0003] rpc error: code = InvalidArgument desc = application spec for test-app is invalid: InvalidSpecError: Unable to generate manifests in bad-path: rpc error: code = Unknown desc = bad-path: app path does not exist

patpicos · 2024-10-30T03:10:58Z

Thanks reza for confirming my thought that the API is more stringent than kubectl apply.

I am curious to know if theres a way to interact with the API to behave like kubectl. If there arent, then crossplane argocd provider wont be able to help with the outcome i seek. I mean the hammering behavior does need addressing.

im going to accelerate plan B and try with kubernetes provider. While waiting to see if an alternative setting i am not aware of exists

blakepettersson · 2024-10-30T09:24:24Z

I think the expectation is to create the app but leave it in not synced state, similar to behavior when you create the app using a manifest

That part I understand. I think the issue is that when submitting to the K8s API, K8s doesn't do any validation of the object other than that it adheres to the given CRD. The validation of the application happens async, when the application-controller picks up the application from k8s.

With the CLI, by default the object will be validated by the Argo CD API. With the CLI this should be able to be worked around by using --validate false.

argocd app create test-app --repo  https://github.com/argoproj/argocd-example-apps.git --path bad-path --dest-namespace default --dest-server https://kubernetes.default.svc  --validate false

In order for the Argo CD Crossplane Provider to provide the same behavior, it would need to pass Validate: false when performing an ApplicationCreateRequest.

blakepettersson · 2024-10-30T09:25:53Z

Update, it is indeed hammering (2-3x per second). I also see that the provider is issueing a list command which is causing a panic

time="2024-10-29T20:10:24Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:24Z" grpc.time_ms=1.054 span.kind=server system=grpc
time="2024-10-29T20:10:24Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1728593133,\"iss\":\"argocd\",\"jti\":\"1a61da56-7ebf-428e-8f2e-d4bea7206c92\",\"nbf\":1728593133,\"sub\":\"crossplane-provider\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:24Z" span.kind=server system=grpc

time="2024-10-29T20:10:25Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for mt-ndc-dev-jzsrn-mt-ndc-us-east-1-mt-ndc-chf-cicd-chf-cnf is invalid: InvalidSpecError: Unable to generate manifests in us-east-1/mt-ndc-chf-cicd/workloads/chf: rpc error: code = Unknown desc = Manifest generation error (cached): us-east-1/mt-ndc-chf-cicd/workloads/chf: app path does not exist" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:24Z" grpc.time_ms=627.098 span.kind=server system=grpc
time="2024-10-29T20:10:25Z" level=info msg="received unary call /application.ApplicationService/List" grpc.method=List grpc.request.claims="{\"iat\":1728593133,\"iss\":\"argocd\",\"jti\":\"1a61da56-7ebf-428e-8f2e-d4bea7206c92\",\"nbf\":1728593133,\"sub\":\"crossplane-provider\"}" grpc.request.content="name:\"nk-rdc-dev-ghshh-nk-rdc-us-east-1-nk-rdc-amf-cicd-demo-amf-cnf\" " grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:25Z" span.kind=server system=grpc

time="2024-10-29T20:10:25Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:25Z" grpc.time_ms=0.568 span.kind=server system=grpc
time="2024-10-29T20:10:25Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1728593133,\"iss\":\"argocd\",\"jti\":\"1a61da56-7ebf-428e-8f2e-d4bea7206c92\",\"nbf\":1728593133,\"sub\":\"crossplane-provider\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2024-10-29T20:10:25Z" span.kind=server system=grpc

That one is likely caused by the Argo CD Provider having a really old version of Argo CD bundled with it. crossplane-contrib/provider-argocd#210 hopefully fixes that.

patpicos · 2024-10-30T14:27:44Z

I think the expectation is to create the app but leave it in not synced state, similar to behavior when you create the app using a manifest

That part I understand. I think the issue is that when submitting to the K8s API, K8s doesn't do any validation of the object other than that it adheres to the given CRD. The validation of the application happens async, when the application-controller picks up the application from k8s.

With the CLI, by default the object will be validated by the Argo CD API. With the CLI this should be able to be worked around by using --validate false.
argocd app create test-app --repo  https://github.com/argoproj/argocd-example-apps.git --path bad-path --dest-namespace default --dest-server https://kubernetes.default.svc  --validate false
In order for the Argo CD Crossplane Provider to provide the same behavior, it would need to pass Validate: false when performing an ApplicationCreateRequest.

Is this the same as the syncOptions below or something different with same name? ( i have the syncOption defined as such)

    syncOptions:
    - Validate=false

blakepettersson · 2024-10-30T19:36:10Z

Is this the same as the syncOptions below or something different with same name? ( i have the syncOption defined as such)

Not exactly - they do the same thing but for different objects. --validate false on the CLI (or Validate: false in the gRPC API) will disable validation for the Application object itself, whereas Validate=false will disable validation for the manifest(s) which pertains to the application.

patpicos · 2024-10-30T19:46:47Z

Thanks Blake. I found the details in the ArgoCD api reference as well. I will close this ticket since ArgoCD exposes the flag, the crossplane-provider does not make it available. I will continue chasing on that side.
crossplane-contrib/provider-argocd#209 (comment)

patpicos added the bug Something isn't working label Oct 29, 2024

patpicos mentioned this issue Oct 29, 2024

ArgoCD Application creation fails when referenced repository/path does not exists crossplane-contrib/provider-argocd#209

Open

patpicos closed this as completed Oct 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Behavior of creating Argo Application thru API (crossplane) vs. Kubectl Apply #20580

Behavior of creating Argo Application thru API (crossplane) vs. Kubectl Apply #20580

patpicos commented Oct 29, 2024 •

edited

Loading

patpicos commented Oct 29, 2024

blakepettersson commented Oct 29, 2024

patpicos commented Oct 29, 2024

patpicos commented Oct 29, 2024 •

edited

Loading

r-nasiri commented Oct 30, 2024

patpicos commented Oct 30, 2024

blakepettersson commented Oct 30, 2024 •

edited

Loading

blakepettersson commented Oct 30, 2024

patpicos commented Oct 30, 2024

blakepettersson commented Oct 30, 2024

patpicos commented Oct 30, 2024

Behavior of creating Argo Application thru API (crossplane) vs. Kubectl Apply #20580

Behavior of creating Argo Application thru API (crossplane) vs. Kubectl Apply #20580

Comments

patpicos commented Oct 29, 2024 • edited Loading

patpicos commented Oct 29, 2024

blakepettersson commented Oct 29, 2024

patpicos commented Oct 29, 2024

patpicos commented Oct 29, 2024 • edited Loading

r-nasiri commented Oct 30, 2024

patpicos commented Oct 30, 2024

blakepettersson commented Oct 30, 2024 • edited Loading

blakepettersson commented Oct 30, 2024

patpicos commented Oct 30, 2024

blakepettersson commented Oct 30, 2024

patpicos commented Oct 30, 2024

patpicos commented Oct 29, 2024 •

edited

Loading

patpicos commented Oct 29, 2024 •

edited

Loading

blakepettersson commented Oct 30, 2024 •

edited

Loading