Issues pulling XRAY Daemon image from Docker.io

[RafalJachimczyk]

Hi,

Our Terraform image dependency is defined like so :

  sidecar_definitions = [{
    name : "xray-daemon"
    image: "amazon/aws-xray-daemon"
    essential : false
    portMappings : [
      {
        protocol : "tcp"
        containerPort : 2000
      },
      {
        protocol : "udp"
        containerPort : 2000
      }
    ]

    logConfiguration : {
      logDriver : "awslogs"
      options : {
        awslogs-group : aws_cloudwatch_log_group.xraydaemon.name
        awslogs-region : var.aws_region
        awslogs-stream-prefix : "amazon/aws-xray-daemon"
      }
    }
  }]

When we deploy this in AWS cloud we're getting an error indicating that Docker.io image is not available (or at least that the response code for the request which is trying to pull it was not correct).

CannotPullContainerError: inspect image has been retried 5 time(s): httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/amazon/aws-xray-daemon/manifests/sha256:e0a815332a9d64c0a1949ae755ed33907c37a4782713009325ea32ab9221...

The error is copied from ECS console.

What is the issue ? Is docker.io having problems ? Or AWS unpublished the image ?

Best Regards,
Rafal

[willarmiros]

Hi @RafalJachimczyk,

The daemon with that hash does exist on Dockerhub: https://hub.docker.com/layers/amazon/aws-xray-daemon/latest/images/sha256-e0a815332a9d64c0a1949ae755ed33907c37a4782713009325ea32ab9221dd28?context=explore

It might be a transient error with Docker or a permissions issue, but the image should be available.

[CarlosDomingues]

We are having the same issue since yesterday. We use an internal AWS CDK lib to deploy applications with our stuff configured correctly (X-Ray, AppMesh, CloudWatch, etc...) and it stopped working.

Today we decided to use the ECR Public image to deploy a new app, but it seems the images are different and some modifications are needed.

EDIT 01: I was able to pull this image from my local PC: docker pull amazon/aws-xray-daemon:3.2.0. That said, it took a really long time to even start the download. Maybe ECS is giving up too quickly?

EDIT 02 When using public.ecr.aws/xray/aws-xray-daemon:3.3.3 ECS was complaining it could not find /usr/bin/xray. After removing the entrypoint (it was configured like so: ["/usr/bin/xray", "-t", "0.0.0.0:2000", "-b", "0.0.0.0:2000"], as explained in the docs), the container actually started, but it seems it cant get its instance metadata:

caused by: Get "http://169.254.169.254/latest/meta-data/instance-id": dial tcp 169.254.169.254:80: 
connect: invalid argument

EDIT 03 Since we can access the docker hub repo, we opted for just pulling it and pushing it to a private ECR. That ended up working fine.

[patmooney]

I'm seeing the same issue as of 02/02/2022.

EDIT: tried again this morning 03/02/2022 and seems to be working again 👍