Image swap registry when using ECR cross region?

[jetersen]

We currently rely on GitOps for our deployments so changing the image registry for our US located deployment in Git is less interesting for checking for updates.

However running image swapper for our US Cluster we would like to swap our EU registry out with our registry in the US region that uses cross region replica. How can I achieve this in the configuration options?

The docs are not very clear on cross region and does not currently seem to support multiple filters and targets per filter.

I would potentially also like to have multiple sources based on different filters and targets with different lifecycle policy.
Instead of running two image swappers.

Cause I would like to copy all non ECR images to ECR and only keep a few images around.
While for images using our ECR registry we want different rules.

This is our current config:

logLevel: info
logFormat: json
imageSwapPolicy: always
imageCopyPolicy: delayed
imageCopyDeadline: 60s

source:

  filters:
  - jmespath: "contains(container.image, '.dkr.ecr.') && contains(container.image, '.amazonaws.com')"

target:
  type: aws
  aws:
    accountId: "snip"
    region: "eu-west-1"
    ecrOptions:
      accessPolicy: |
        {
          "Statement": [
            {
              "Sid": "AllowCrossAccountPull",
              "Effect": "Allow",
              "Principal": {
                "AWS": "*"
              },
              "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability"
              ],
              "Condition": {
                "StringEquals": {
                  "aws:PrincipalOrgID": "SNIP"
                }
              }
            }
          ],
          "Version": "2008-10-17"
        }
      lifecyclePolicy: |
        {
          "rules": [
            {
              "rulePriority": 1,
              "description": "Rule 1",
              "selection": {
                "tagStatus": "untagged",
                "countType": "sinceImagePushed",
                "countUnit": "days",
                "countNumber": 1
              },
              "action": {
                "type": "expire"
              }
            },
            {
              "rulePriority": 2,
              "description": "Rule 2",
              "selection": {
                "tagStatus": "any",
                "countType": "imageCountMoreThan",
                "countNumber": 5
              },
              "action": {
                "type": "expire"
              }
            }
          ]
        }

[estahn]

@jetersen Just to summarise for my understanding. You would like to change the source address based on certain criteria?

Example:

• Source: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/mservice:42
• k8s-image-swapper Result: 123456789012.dkr.ecr.us-east-1.amazonaws.com/mservice:42

In case the image is missing in US East (us-east-1), would that be written to ECR eu-west-1 from the US cluster?

[jetersen]

Yes that is exactly what I want to achieve in addition to delayed copying open source images in the hope that eu to us replication already has copied image

The image is not missing as it would be replicated by AWS cross region replication.

So eu-west-1 registry is the source of truth and us-east-1 is our replica registry.

Potentially if image swapper could check for us-east-1 availability if not simply do not mutate and once next pod rolls around it would then start mutating because the image got replicated

[estahn]

Got it. I think this would partially work as follows:

Cluster US:

• ImageSwapPolicy: exists
• ImageCopyPolicy: none

What is missing is some way to alter the source address, e.g.

source:
  filters:
  - jmespath: "contains(container.image, '.dkr.ecr.') && contains(container.image, '.amazonaws.com')"

  preprocessor:
  - replace:
       from: dkr.ecr.eu-west-1.amazonaws.com
       to: dkr.ecr.us-east-1.amazonaws.com

Do you think that would work for your use-case?

[jetersen]

Oh ya a preprocessor step you could as long as it follows the normal path to check if the modified source image exists and if it does do nothing. I think that should work for my use case 👏
That should help with most of the rate limiting issues we seen.

[wosiu]

@jetersen I think this: https://github.com/jainishshah17/tugger?tab=readme-ov-file#configure
does exactly what you need.
I look at k8s-image-swapper as a tool for mimicking pull-through cache, because it pulls and pushes image into regional pull-through cache. What you need is just mutating a registry, which I think the linked admission webhook is better with its flexibility.

[jetersen]

But image swapper is an admission webhook already 😅 Why do I need another?