Authenticating to AWS EKS Cluster

[jseiser]

We are having problem getting Atlantis to be able to run TF commands against the EKS cluster.

❯ kubectl get configmap -n kube-system aws-auth -o yaml
apiVersion: v1
data:
  mapAccounts: |
    []
  mapRoles: |
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws-us-gov:iam::xxx:role/role-atlantis-prod"
      "username": "atlantis-admin"

Pod is deployed with a serviceAccount

  serviceAccount: atlantis-sa

That Service account is wired up for IRSA

❯ kubectl get serviceaccount atlantis-sa -n atlantis -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-us-gov:iam::xxx:role/role-prod-atlantis

The pod has all the env vars I would expect

_apt@atlantis-0:/$ printenv | grep AWS
AWS_DEFAULT_REGION=us-gov-west-1
AWS_REGION=us-gov-west-1
AWS_ROLE_ARN=arn:aws-us-gov:iam::xxx:role/role-prod-atlantis
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional

it works at a basic level

atlantis@atlantis-0:~/.aws$ aws s3 ls
2022-06-30 15:46:21 cloudtrail-global-xxxxx
2022-09-26 15:33:38 xxx-qa-xxxx

Our Gitlab instance is deployed the same way, and it works. So I dont think its an issue on the aws/auth side

Our dockerfile

FROM ghcr.io/runatlantis/atlantis:v0.22.3-debian

RUN sh -c 'curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o "awscliv2.zip" && unzip awscliv2.zip && ./aws/install && rm -rf ./aws awscliv2.zip && aws --version'

When i hop into the pod, i can run aws CLI, but generating a token fails

_apt@atlantis-0:/$ aws --version
aws-cli/2.9.17 Python/3.9.11 Linux/5.4.226-129.415.amzn2.x86_64 exe/x86_64.debian.11 prompt/off
_apt@atlantis-0:/$ aws eks get-token --cluster-name eks-cluster-prod

[Errno 13] Permission denied: '/nonexistent'

Makes sense, since this chart runs you as a user, which doesnt have a home dir. Im not sure why that is, but its expected.

So i tried to run as the atlantis user, and that lets you generate a token

statefulSet:
  securityContext:
    fsGroup: 1000
    runAsUser: 1000
    fsGroupChangePolicy: "OnRootMismatch"

atlantis@atlantis-0:/$ whoami
atlantis
atlantis@atlantis-0:/$ aws eks get-token --cluster-name eks-cluster-prod
{"kind": "ExecCredential", "apiVersion": "client.authentication.k8s.io/v1beta1", "spec": {}, "status": {"expirationTimestamp": "2023-01-25T21:51:43Z", "token": "k8s-aws-v1.aHR0cHM6Lyxxxxxxx"}}

but no matter what, I get. Which i really doubt, since we have already confirmed we have a SA, that has access to AWS s3, and that it exists in the aws-auth configmap.

╷
│ Error: Unauthorized
│ 
│   with module.eks.kubernetes_config_map_v1_data.aws_auth[0],
│   on .terraform/modules/eks/[main.tf](http://main.tf/) line 527, in resource "kubernetes_config_map_v1_data" "aws_auth":
│  527: resource "kubernetes_config_map_v1_data" "aws_auth" {
│ 
╵```

[jseiser]

even trying kubectl manually, doesnt work

Get token/config

atlantis@atlantis-0:/$ aws eks update-kubeconfig --name eks-prod-cluster
Added new context arn:aws-us-gov:eks:us-gov-west-1:xxxx:cluster/eks-prod-cluster to /home/atlantis/.kube/config

get kubectl

atlantis@atlantis-0:~$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   138  100   138    0     0   1682      0 --:--:-- --:--:-- --:--:--  1682
100 45.7M  100 45.7M    0     0  45.3M      0  0:00:01  0:00:01 --:--:-- 77.4M
atlantis@atlantis-0:~$ chmod +x kubectl

do something

atlantis@atlantis-0:~$ ./kubectl get namespaces
E0125 21:55:02.745655      67 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0125 21:55:03.446317      67 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0125 21:55:04.244511      67 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0125 21:55:04.976649      67 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0125 21:55:05.713037      67 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

[nitrocode]

If you're getting the same issue with kubectl then it doesn't sound like an Atlantis issue. This sounds like an issue in the eks auth config map permissions

[jseiser]

I'm not sure that's the case. We can see the iam role is in the config map, we can see the service account has the iam role and that the pod has the service account. Everything else works with that setup. I will launch an ephemeral container with the same setup so I can isolate this to the Atlantis container itself.

…

On Wed, Jan 25, 2023, 7:02 PM nitrocode ***@***.***> wrote: If you're getting the same issue with kubectl then it doesn't sound like an Atlantis issue. This sounds like an issue in the eks auth config map permissions — Reply to this email directly, view it on GitHub <#257 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFBNZ5357DPX3FBWT3VGDLWUG5IFANCNFSM6AAAAAAUG2EUTE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jseiser]

@nitrocode

I want to thank you for being a much better person than I am, because I am an idiot.

AWS_ROLE_ARN=arn:aws-us-gov:iam::xxx:role/role-prod-atlantis ~= "rolearn": "arn:aws-us-gov:iam::xxx:role/role-atlantis-prod"

[nitrocode]

Please don't say that because it cannot be true due to all of us working with very complicated systems. Thank you for running through all the iterations to isolate your issue and for closing the loop on this. 😄

Happy your issue is solved!