Cross-Account IRSA with Role Chaining

[aljoshare]

Hello everyone,

I try to access a S3 bucket from a different account via Cross-Account IRSA and Role Chaining. If I try it from an aws-cli container with the same configuration (same service account with annotation, same ~/.aws/config with chained roles) everything works fine. I can do aws s3 ls BUCKETNAME without any further configuration needed. If I apply the same configuration to the ingester, it never passes the sanity-check.

I saw that you are using the thanos.io/objstore library and tried to enable aws_sdk_auth but it doesn't solve the problem. Do you have any idea why it doesn't work?

Values.yml:

ingester:
  extraVolumes:
     - name: aws-config
       configMap:
         name: mimir-aws-config
  extraVolumeMounts:
     - name: aws-config
       mountPath: "/root/.aws"
       readOnly: true
  env:
    - name: "AWS_PROFILE"
      value: "default"
    - name: "AWS_REGION"
      value: "eu-central-1"
    - name: "AWS_CONFIG_FILE"
      value: "/root/.aws/config"

serviceAccount:
  create: true
  name: mimir
  annotations:
    app.kubernetes.io/name: "mimir"
    eks.amazonaws.com/role-arn: arn:aws:iam::[REDACTED]:role/grafana-mimir-s3-assume-role


mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          endpoint: s3.eu-central-1.amazonaws.com
          region: eu-central-1
          aws_sdk_auth: true // Added but it doesn't help
    blocks_storage:
      s3:
        bucket_name: BUCKETNAME

AWS config:

    [profile default]
    source_profile = assume-role
    role_arn=arn:aws:iam::[REDACTED]:role/grafana-mimir-s3-role
    
    [profile assume-role]
    web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    role_arn=arn:aws:iam::[REDACTED]:role/grafana-mimir-s3-assume-role

Thank you very much!

[aljoshare]

I tried to assume the role manually and use the access key, access key secret and the session token and it works now. Maybe it's not even the objstore but the AWS SDK which can't handle role chaining properly.

[aljoshare]

Found the issue and created a corresponding PR for it.