I get error decoding CKS cluster installation after upgrade from 4.19 into 4.21

[ahmadamirahmadi007]

2025-11-08 14:59:24,445 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-93:[ctx-714cf072, job-60993/job-60995, ctx-b9d9e8f7]) (logid:eb39427a) Trying to deploy VM [error decoding VM instance {"id":3648,"instanceName":"i-2-3648-VM","state":"Starting","type":"User","uuid":"2ac6d25d-2f3d-40d8-bc0c-acd599154496"}] and details: Plan [error decoding {"_clusterId":null,"_dcId":1,"_physicalNetworkId":null,"_podId":null,"_poolId":null,"migrationPlan":false}]; avoid list [{}] and planner: [null].

creating instances separately is ok, but when I create a kubernetes cluster, it doesn't start worker or control node instances inside it.
Is there a problem somewhere in the database?

[vishesh92]

@ahmadamirahmadi007 Can you share more logs? It's most likely some configuration issue. You can ignore the "error decoding" part. I think it's a logging issue and not related to your problem.

[kiranchavala]

@ahmadamirahmadi007

Are you launching cks in an existing network or a new network

Could you please try restarting the network with cleanup option if its a existing network and check again by deploying a new cks cluser

[ahmadamirahmadi007]

@kiranchavala @vishesh92
I have used the new network and the problem is still there. But what I have noticed is that this problem is happening to the admin user who is in the root domain. When I create a new user who is a member of domain-admin, the Kuber cluster installs and starts up without any problems. I am getting the following errors while installing the Kuber cluster with the admin user.

2025-11-15 12:39:42,438 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-66:[ctx-c1a8d6b5, job-65305/job-65307, ctx-3fc067ea]) (logid:509bf6b1) Trying to deploy VM [error decoding VM instance {"id":3705,"instanceName":"i-2-3705-VM","state":"Starting","type":"User","uuid":"57ef2be5-ea4c-431f-a1fc-d0f3c47ff91d"}] and details: Plan [error decoding {"_clusterId":null,"_dcId":1,"_physicalNetworkId":null,"_podId":null,"_poolId":null,"migrationPlan":false}]; avoid list [{}] and planner: [null].

2025-11-15 12:39:42,636 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-66:[ctx-c1a8d6b5, job-65305/job-65307, ctx-3fc067ea]) (logid:509bf6b1) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3705,"instanceName":"i-2-3705-VM","state":"Starting","type":"User","uuid":"57ef2be5-ea4c-431f-a1fc-d0f3c47ff91d"}Scope=interface com.cloud.dc.DataCenter; id=1

2025-11-15 12:39:42,636 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-66:[ctx-c1a8d6b5, job-65305/job-65307]) (logid:509bf6b1) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAOeXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA","cmdVersion":0,"completeMsid":null,"created":"Sat Nov 15 12:39:40 UTC 2025","id":65307,"initMsid":345051312967,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"fa395beb-ae36-4459-bb8b-f8febb758bb4"}, job origin: 65305 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3705,"instanceName":"i-2-3705-VM","state":"Starting","type":"User","uuid":"57ef2be5-ea4c-431f-a1fc-d0f3c47ff91d"}Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1451)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at jdk.internal.reflect.GeneratedMethodAccessor1145.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2025-11-15 12:39:42,895 ERROR [c.c.k.c.a.KubernetesClusterStartWorker] (API-Job-Executor-80:[ctx-617b3ff2, job-65305, ctx-9845117c]) (logid:509bf6b1) Provisioning the control VM failed in the Kubernetes cluster : admin-kuber com.cloud.exception.ManagementServerException: Failed to start VM in the Kubernetes cluster : admin-kuber
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.startKubernetesVM(KubernetesClusterResourceModifierActionWorker.java:343)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.provisionKubernetesClusterControlVm(KubernetesClusterStartWorker.java:520)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startKubernetesClusterOnCreate(KubernetesClusterStartWorker.java:790)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1790)

[DaanHoogland]

I have used the new network and the problem is still there. But what I have noticed is that this problem is happening to the admin user who is in the root domain. When I create a new user who is a member of domain-admin, the Kuber cluster installs and starts up without any problems. I am getting the following errors while installing the Kuber cluster with the admin user.

Can you check the limits for either user @ahmadamirahmadi007 ?

[prashanthr2]

@ahmadamirahmadi007 Can you share full management server log that covers the time of the issue