Image push with Kaniko stuck on ContainersNotReady containers with unready status: [polyaxon-main polyaxon-sidecar]

[sathoune]

This question was resolved and discussed on Polyaxon Slack. Posting for visibility if someone stumbles upon the same issue.

Given setup:

• Docker-registry provider: Amazon Elastic Container Registry (ECR)
• Polyaxon version: 1.7.3 CE
• Deployed with Kubernetes on AWS

And credentials setup from Kaniko github: Pushing to Amazon ECR

And Kaniko integration in polyaxon-config.yml:

connections:
  - name: docker-registry
    kind: registry
    description: "aws docker repository"
    schema:
      url: https://ID.dkr.ecr.SOME-REGION.amazonaws.com
    secret:
      name: docker-conf
      mountPath: /kaniko/.docker

And polyaxonfile.yml from polyaxon example:

version: 1.1
kind: operation
name: build
params:
  destination:
    connection: docker-registry
    value: polyaxon-examples:ml
runPatch:
  init:
  - dockerfile:
      image: "tensorflow/tensorflow:2.0.1-py3"
      run:
      - 'pip3 install --no-cache-dir -U polyaxon["polyboard","polytune"]'
      langEnv: 'en_US.UTF-8'
hubRef: kaniko

Then warning was raised:

and job would be stuck like this until manually stopped.

TYPE     STATUS    REASON              MESSAGE                                                           LAST_UPDATE_TIME    LAST_TRANSITION_TIME
-------  --------  ------------------  ----------------------------------------------------------------  ------------------  ----------------------
warning  True      ContainersNotReady  containers with unready status: [polyaxon-main polyaxon-sidecar]  a few seconds ago   a few seconds ago

[sathoune]

Resolution:

The issue was with aws-secret type.

At first we had the type of secret that expires every 12 hours.
Changing it to one that does not expire allowed successful connection and push of built image.