Polyaxon CE fresh deployment never finishes

[polyaxon-team]

From slack

Hi, I’m trying to deploy Polyaxon CE with helm/argocd and the polyaxon-api keeps logging in loop:

System check identified some issues:

Preparing...

and the pod never becomes ready.
Any idea what could cause that?

More

It seems the polyaxon helm chart installation always fails with the message:

Error: timed out waiting for the condition
This is what I see:

polyaxon-polyaxon-api-d8d7c8b5f-spsb2           1/1     Running   0          3d5h
polyaxon-polyaxon-api-dbb84b79c-6jqm9           0/1     Running   2          13m

The polyaxon api pods take a long time to become ready. It fails liveness probe:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  15m                   default-scheduler  Successfully assigned polyaxon/polyaxon-polyaxon-api-dbb84b79c-6jqm9 to ip-192-168-165-30.us-east-2.compute.internal
  Normal   Pulling    14m                   kubelet            Pulling image "polyaxon/polyaxon-api:xx"
  Normal   Pulled     14m                   kubelet            Successfully pulled image "polyaxon/polyaxon-api:xx" in 33.387487209s
  Normal   Created    14m                   kubelet            Created container polyaxon-api
  Normal   Started    14m                   kubelet            Started container polyaxon-api
  Normal   Killing    8m49s                 kubelet            Container polyaxon-api failed liveness probe, will be restarted
  Warning  Unhealthy  8m37s (x10 over 13m)  kubelet            Readiness probe failed: Get "[http://192.168.184.124:80/healthz/](http://192.168.184.124/healthz/)": dial tcp 192.168.184.124:80: connect: connection refused
  Normal   Pulled     8m19s                 kubelet            Container image "polyaxon/polyaxon-api:1.9.5" already present on machine
  Warning  Unhealthy  4m49s (x16 over 13m)  kubelet            Liveness probe failed: Get "[http://192.168.184.124:80/healthz/](http://192.168.184.124/healthz/)": dial tcp 192.168.184.124:80: connect: connection refused

[polyaxon-team]

The firts step is to cehck if you are using the --wait flag with Helm, if yes the deployment will not pass because there's helm hook.

The reason of the deadlock: helm waits forever for API to be healthy and the API is waiting for the hook to initialize the database.

More info about the helm deadlock issue: helm/helm#5118

For ArgoCD, the flag is set to true automatically: argoproj/argo-cd#6880

For the Terraform provider for helm release, the flag is set to true by default: https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release#wait and should be set it to false.