kubernetes cluster does not create after upgrade

[ahmadamirahmadi007]

problem

I upgraded my cs from 4.19 to 4.21. I faced a problem with creating a Kubernetes cluster. i get error below during install cluster. i can create instance but i can not create k8s cluster.

2025-11-08 14:59:24,445 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-93:[ctx-714cf072, job-60993/job-60995, ctx-b9d9e8f7]) (logid:eb39427a) Trying to deploy VM [error decoding VM instance {"id":3648,"instanceName":"i-2-3648-VM","state":"Starting","type":"User","uuid":"2ac6d25d-2f3d-40d8-bc0c-acd599154496"}] and details: Plan [error decoding {"_clusterId":null,"_dcId":1,"_physicalNetworkId":null,"_podId":null,"_poolId":null,"migrationPlan":false}]; avoid list [{}] and planner: [null].

2025-11-08 14:59:24,627 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-93:[ctx-714cf072, job-60993/job-60995, ctx-b9d9e8f7]) (logid:eb39427a) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3648,"instanceName":"i-2-3648-VM","state":"Starting","type":"User","uuid":"2ac6d25d-2f3d-40d8-bc0c-acd599154496"}Scope=interface com.cloud.dc.DataCenter; id=1

2025-11-08 14:59:24,628 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-93:[ctx-714cf072, job-60993/job-60995]) (logid:eb39427a) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAOQHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA","cmdVersion":0,"completeMsid":null,"created":"Sat Nov 08 14:59:23 UTC 2025","id":60995,"initMsid":345051312967,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"13a1ef61-3bf1-4d5d-88fc-77b1105ee6a2"}, job origin: 60993 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3648,"instanceName":"i-2-3648-VM","state":"Starting","type":"User","uuid":"2ac6d25d-2f3d-40d8-bc0c-acd599154496"}Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1451)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at jdk.internal.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-11-08 14:59:24,778 ERROR [c.c.k.c.a.KubernetesClusterStartWorker] (API-Job-Executor-30:[ctx-424ee4ff, job-60993, ctx-a587dac5]) (logid:eb39427a) Provisioning the control VM failed in the Kubernetes cluster : new com.cloud.exception.ManagementServerException: Failed to start VM in the Kubernetes cluster : new
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.startKubernetesVM(KubernetesClusterResourceModifierActionWorker.java:343)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.provisionKubernetesClusterControlVm(KubernetesClusterStartWorker.java:520)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startKubernetesClusterOnCreate(KubernetesClusterStartWorker.java:790)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1790)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1703)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy534.startKubernetesCluster(Unknown Source)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd.execute(CreateKubernetesClusterCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":3648,"instanceName":"i-2-3648-VM","state":"Starting","type":"User","uuid":"2ac6d25d-2f3d-40d8-bc0c-acd599154496"}Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1451)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at jdk.internal.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)

2025-11-08 14:59:24,881 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-30:[ctx-424ee4ff, job-60993]) (logid:eb39427a) Complete async job-60993, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Provisioning the control VM failed in the Kubernetes cluster : new"}

versions

ACS 4.21
3 KVM hosts

The steps to reproduce the bug

upgrade ACS from 4.19.1.3 into 4.21
create k8s cluster with error

What to do about it?

I want to create a cluster properly, like before the upgrade

[DaanHoogland]

@ahmadamirahmadi007 , your output seems to indicate there is not enough capacity to deploy the cluster. No deployment plan can be created. Can you check the capacity of your cloud?

[ahmadamirahmadi007]

@DaanHoogland
I checked and I have enough space because I can easily run multiple instances separately from the Kubernetes cluster. Please check this ticket as well.
#12052

[DaanHoogland]

@ahmadamirahmadi007 search your logs up from the error using any of the terms job-60993 and logid:eb39427a. you will either find the root cause in a DEBUG message or a message saying that an empty answer was returned by a backend system, in which case you will need to study the logs of that system.