Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PodidentityAssociation ready state is false #168

Open
ashishvaishno opened this issue Feb 14, 2024 · 5 comments
Open

PodidentityAssociation ready state is false #168

ashishvaishno opened this issue Feb 14, 2024 · 5 comments
Labels
question Further information is requested

Comments

@ashishvaishno
Copy link

ashishvaishno commented Feb 14, 2024

Hello,

I have upgrade my crossplane deployment to xpkg.upbound.io/upbound/provider-aws-eks:v1.0.0. I have started to use Podidentityassociation.

Crossplane was successfully able to create the association and synced, but the ready state is false 😕 Below is the error of the managed resource. I dont have any previous managed resource / already existing association with an association id a-stubassocid123456

async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-stubassocid123456"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: f9cb61ba-2e55-4c0e-a8f7-962bfd8302bc, ResourceInUseException: Association already exists: a-dt1bedfjc72dlwp5n: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: f9cb61ba-2e55-4c0e-a8f7-962bfd8302bc, ResourceInUseException: Association already exists: a-dt1bedfjc72dlwp5n

Screenshot 2024-02-14 at 17 06 04

@candonov
Copy link
Contributor

candonov commented Mar 4, 2024

Hi @ashishvaishno, can you provide the Crossplane manifest(s) you used so I reproduce this?

@ashishvaishno
Copy link
Author

ashishvaishno commented Mar 4, 2024

@candonov Sure. But the issue persists only on few PIA. I tried to clean them up and re-deploy them as well. But its still the same 😢
Composition

# XRD to create the s3 bucket and the associated irsa
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xs3withpodidentitys.crossplane.xyz.com
spec:
  claimNames:
    kind: S3WithPodIdentity
    plural: s3withpodidentitys
  group: crossplane.xyz.com
  names:
    kind: XS3WithPodIdentity
    plural: xs3withpodidentitys
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          properties:
            spec:
              description: S3WithPodIdentitySpec defines the desired state of S3WithPodIdentity
              properties:
                resourceConfig:
                  description: ResourceConfig defines general properties of this AWS resource.
                  properties:
                    deletionPolicy:
                      description: Defaults to Delete
                      enum:
                      - Orphan
                      type: string
                      default: "Orphan"
                    bucketName:
                      description: Name of the bucket
                      type: string
                    serviceAccountName:
                      description: Name of the service account which will also be used to generate the k8s sa
                      type: string
                    serviceNamespace:
                      description: Namespace of the service where its running
                      type: string
                    bucketVersioningStatus:
                      description: Bucket versioning status
                      type: string
                      default: "Disabled"
                      enum:
                      - Disabled
                      - Suspended
                      - Enabled
                    bucketNonCurrentVersionExpirationDays:
                      description: Cleanup Old Objects older than x days
                      type: number
                      default: 90                     
                    bucketCleanupOldObjectStatus:
                      description: Cleanup Old Objects
                      type: string
                      default: "Disabled"
                      enum:
                      - Disabled
                      - Enabled
                    bucketCleanupOldObjectExpirationDays:
                      description: Cleanup Old Objects older than x days
                      type: number
                      default: 180
                    awsBackup:
                      description: Setup awsbackup to exclude for s3 buckets
                      type: string
                      default: "exclude"
                    tags:
                      items:
                        properties:
                          key:
                            type: string
                          value:
                            type: string
                        required:
                        - key
                        - value
                        type: object
                      type: array
                  required:
                  - bucketName
                  - serviceAccountName
                  - serviceNamespace
                  type: object
              required:
              - resourceConfig
              type: object
            status:
              description: S3WithPodIdentityStatus defines the observed state of S3WithPodIdentity
              properties:
                bucketArn:
                  type: string
                bucketVersioningStatus:
                  type: string
                bucketNonCurrentVersionExpirationDays:
                  type: number
                bucketCleanupOldObjectStatus:
                  type: string
                bucketCleanupOldObjectExpirationDays:
                  type: number
                roleArn:
                  type: string
                awsAccountID:
                  type: string
                policyArn:
                  type: string
              type: object
          type: object

XRD :

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: dev.xs3withpodidentity.crossplane.xyz.com
  labels:
    environment: dev
spec:
  compositeTypeRef:
    apiVersion: crossplane.xyz.com/v1alpha1
    kind: XS3WithPodIdentity
  patchSets:
    - name: deletion-policy-field
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.deletionPolicy
          toFieldPath: spec.deletionPolicy
    - name: common-fields
      patches:
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            strategy: string
            string:
              fmt: "aws-provider-config-%s"
          toFieldPath: spec.providerConfigRef.name
    - name: tags
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
          toFieldPath: spec.forProvider.tags.Environment
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.tags.serviceAccountName
    - name: region
      patches:          
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
          toFieldPath: spec.forProvider.region
          transforms:
            - type: map
              map:
                dev: eu-central-1
                stage: eu-west-1
                prod: eu-west-1
    - name: s3-extra-tag
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.awsBackup
          toFieldPath: spec.forProvider.tags.AWSBackup
  resources:
    - name: s3-bucket
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: Bucket
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: tags
        - type: PatchSet
          patchSetName: region
        - type: PatchSet
          patchSetName: s3-extra-tag
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketName
          toFieldPath: metadata.annotations[crossplane.io/external-name]
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketName
          toFieldPath: spec.forProvider.tags.Name
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.arn
          toFieldPath: status.bucketArn
    - name: s3-bucketpublicaccessblock
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketPublicAccessBlock
        spec:
          forProvider:
            blockPublicAcls: true
            blockPublicPolicy: true
            ignorePublicAcls: true
            restrictPublicBuckets: true
            bucketSelector:
              matchControllerRef: true
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
    - name: s3-bucketserversideencryptionconfiguration
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketServerSideEncryptionConfiguration
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            rule:
              - applyServerSideEncryptionByDefault:
                  - sseAlgorithm: AES256
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
    - name: s3-bucketversioning
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketVersioning
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            versioningConfiguration:
              - status: ""
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketVersioningStatus
          toFieldPath: spec.forProvider.versioningConfiguration[0].status
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.versioningConfiguration[0].status
          toFieldPath: status.bucketVersioningStatus
    - name: s3-bucketlifecycleconfiguration
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketLifecycleConfiguration
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            rule:
              - id: "Cleanup of Non Current Versions"
                noncurrentVersionExpiration:
                  - noncurrentDays: 
                status: ""
              - id: "Cleanup of Old Objects"
                expiration:
                  - days: 
                status: ""
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: deletion-policy-field
        - type: PatchSet
          patchSetName: region
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketVersioningStatus
          toFieldPath: spec.forProvider.rule[0].status
          transforms:
            - type: map
              map:
                Disabled: Disabled
                Suspended: Enabled
                Enabled: Enabled
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketNonCurrentVersionExpirationDays
          toFieldPath: spec.forProvider.rule[0].noncurrentVersionExpiration[0].noncurrentDays
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketCleanupOldObjectStatus
          toFieldPath: spec.forProvider.rule[1].status
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.bucketCleanupOldObjectExpirationDays
          toFieldPath: spec.forProvider.rule[1].expiration[0].days
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[0].status
          toFieldPath: status.bucketVersioningStatus
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[0].noncurrentVersionExpiration[0].noncurrentDays
          toFieldPath: status.bucketNonCurrentVersionExpirationDays
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[1].status
          toFieldPath: status.bucketCleanupOldObjectStatus
        - type: ToCompositeFieldPath
          fromFieldPath: spec.forProvider.rule[1].expiration[0].days
          toFieldPath: status.bucketCleanupOldObjectExpirationDays
    - name: iamrole
      base:
        apiVersion: iam.aws.upbound.io/v1beta1
        kind: Role
        spec:
          deletionPolicy: Delete
          forProvider:
            assumeRolePolicy: |
              {
                "Version": "2012-10-17",
                "Statement": [
                  {
                    "Effect": "Allow",
                    "Principal": {
                      "Service": "pods.eks.amazonaws.com"
                    },
                    "Action": [
                      "sts:TagSession",
                      "sts:AssumeRole"
                    ]
                  }
                ]
              }
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: tags
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            strategy: string
            string:
              fmt: "k8s-%s-%s"
          toFieldPath: metadata.annotations[crossplane.io/external-name]
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.arn
          toFieldPath: status.roleArn
    - name: iam-policy
      base: 
        apiVersion: iam.aws.upbound.io/v1beta1
        kind: Policy
        spec:
          deletionPolicy: Delete
          forProvider:
            description: ""
            policy: ""
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: PatchSet
          patchSetName: tags
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            strategy: string
            string:
              fmt: "k8s-%s-%s"
          toFieldPath: metadata.annotations[crossplane.io/external-name]
        - type: ToCompositeFieldPath
          fromFieldPath: status.atProvider.arn
          toFieldPath: status.policyArn
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: spec.resourceConfig.bucketName
            strategy: string
            string:
              fmt: "IAM Policy for bucket %s created by crossplane"
          toFieldPath: spec.forProvider.description
        - type: CombineFromComposite
          toFieldPath: spec.forProvider.policy
          policy:
            fromFieldPath: Required
          combine:
            variables:
            - fromFieldPath: spec.resourceConfig.bucketName
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            - fromFieldPath: spec.resourceConfig.bucketName
            - fromFieldPath: spec.resourceConfig.serviceAccountName
            strategy: string
            string:
              fmt: |
                { 
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Action": "s3:ListBucket",
                      "Effect": "Allow",
                      "Resource": [
                          "arn:aws:s3:::%s"
                      ],
                      "Condition": {
                        "StringLike": {
                          "aws:PrincipalTag/kubernetes-pod-name": "%s*"
                        }
                      }
                    },
                    {
                      "Action": [
                        "s3:ListObjects",
                        "s3:PutObject",
                        "s3:GetObject",
                        "s3:DeleteObject"
                      ],
                      "Effect": "Allow",
                      "Resource": [
                          "arn:aws:s3:::%s/*"
                      ],
                      "Condition": {
                        "StringLike": {
                          "aws:PrincipalTag/kubernetes-pod-name": "%s*"
                        }
                      }
                    }
                  ]
                }
    - name: policy-attachment-1
      base:
        apiVersion: iam.aws.upbound.io/v1beta1
        kind: RolePolicyAttachment
        spec:
          deletionPolicy: Delete
          forProvider:
            policyArnSelector:
              matchControllerRef: true
            roleSelector:
              matchControllerRef: true
      patches:
        - type: PatchSet
          patchSetName: common-fields
    - name: pod-identity-association
      base:
        apiVersion: eks.aws.upbound.io/v1beta1
        kind: PodIdentityAssociation
        spec:
          deletionPolicy: Delete
          forProvider:
            clusterName: "go-main-dev"
            region: "eu-central-1"
            namespace: ""
            serviceAccount: ""
            roleArnSelector:
              matchControllerRef: true
      patches:
        - type: PatchSet
          patchSetName: common-fields
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceNamespace
          toFieldPath: spec.forProvider.namespace
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.serviceAccount
    - name: service-account
      base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        spec:
          deletionPolicy: Delete
          forProvider:
            manifest:
              apiVersion: v1
              kind: ServiceAccount
              metadata:
                name: ""
                namespace: ""
              imagePullSecrets:
                - name: registrypullsecret
      patches:
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            strategy: string
            string:
              fmt: "kubernetes-provider-config-%s"
          toFieldPath: spec.providerConfigRef.name
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceNamespace
          toFieldPath: spec.forProvider.manifest.metadata.namespace
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.manifest.metadata.name
    - name: remote-configmap
      base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        spec:
          deletionPolicy: Delete
          forProvider:
            manifest:
              apiVersion: v1
              kind: ConfigMap
              metadata:
                name: ""
                namespace: ""
              data:
                roleArn: ""
                policyArn: ""
                bucketArn: ""
                serviceAccountName: ""
                region: ""
                bucketVersioningStatus: ""
      patches:
        - type: CombineFromComposite
          combine:
            variables:
            - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
            strategy: string
            string:
              fmt: "kubernetes-provider-config-%s"
          toFieldPath: spec.providerConfigRef.name
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceNamespace
          toFieldPath: spec.forProvider.manifest.metadata.namespace
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.manifest.metadata.name
        - type: FromCompositeFieldPath
          fromFieldPath: status.roleArn
          toFieldPath: spec.forProvider.manifest.data.roleArn
        - type: FromCompositeFieldPath
          fromFieldPath: status.policyArn
          toFieldPath: spec.forProvider.manifest.data.policyArn
        - type: FromCompositeFieldPath
          fromFieldPath: status.bucketArn
          toFieldPath: spec.forProvider.manifest.data.bucketArn
        - type: FromCompositeFieldPath
          fromFieldPath: spec.resourceConfig.serviceAccountName
          toFieldPath: spec.forProvider.manifest.data.serviceAccountName
        - type: FromCompositeFieldPath
          fromFieldPath: status.bucketVersioningStatus
          toFieldPath: spec.forProvider.manifest.data.bucketVersioningStatus
        - type: FromCompositeFieldPath
          fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
          toFieldPath: spec.forProvider.manifest.data.region
          transforms:
            - type: map
              map:
                dev: eu-central-1
                stage: eu-west-1
                prod: eu-west-1

Claim :

---
apiVersion: crossplane.xyz.com/v1alpha1
kind: S3WithPodIdentity
metadata:
  name: test-s3with-podidentity
  namespace: dev
spec:
  compositionSelector:
    matchLabels:
      environment: dev
  resourceConfig:
    bucketName: test-s3with-podidentity
    serviceAccountName:service-account-name
    serviceNamespace: default

Error :

    Message:               async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-dzwb0z8xhym98oebr"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 428d429c-75f8-4d31-bf88-cdfa1f756370, ResourceInUseException: Association already exists: a-dzwb0z8xhym98oebr: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 428d429c-75f8-4d31-bf88-cdfa1f756370, ResourceInUseException: Association already exists: a-dzwb0z8xhym98oebr

On AWS is do see the association a-dzwb0z8xhym98oebr created by crossplane. Not sure where is the drift 🤷‍♂️

@ashishvaishno
Copy link
Author

@candonov Were you able to test this out?

@candonov
Copy link
Contributor

candonov commented Jun 4, 2024

Hi @ashishvaishno , I tried testing but the provider names are different than the ones in this repository. I know it's been a while, did you get this working or should we dig in?

@ashishvaishno
Copy link
Author

ashishvaishno commented Aug 5, 2024

@candonov Sorry for the late reply. This is still an issue. It also complains for already existing resources

async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-wlcylmg69wzd65sp9"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 558cbce6-04a6-4e15-bf01-a047e2c3ffb3, ResourceInUseException: Association already exists: a-wlcylmg69wzd65sp9: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 558cbce6-04a6-4e15-bf01-a047e2c3ffb3, ResourceInUseException: Association already exists: a-wlcylmg69wzd65sp9
  • Can I force crossplane reconciliation if the PIA already exists somehow?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants