Having issues on activating the Advanced Zone on 4.21.0 #11993

n4l5u0r · 2025-11-05T18:21:27Z

n4l5u0r
Nov 5, 2025

Trying to deploy 4.21.0 - same deployment used to work perfectly on 4.20.0

mgmt network : 10.10.0.0/20 - VLAN
storage network : 10.20.0.0/20 - VLAN
public network : 10.40.0.0/20 - VLAN
guest network : 10.30.0.0/20 - VXLAN

Network topology is the following:

mgmt servers : 10.10.0.0/20 / 10.20.0.0/20
- mgmt-1 : 10.X0.0.3
- mgmt-2 : 10.X0.0.4
KVM servers: 10.10.0.0/20, 10.20.0.0/20, 10.40.0.0/20
- kvm-1 : 10.X0.0.6
- kvm-2 : 10.X0.0.7

When I add an Advanced Zone on a fresh instal of 4.21.0 with 2 mgmt servers and 2 kvm servers I have this issue during the creation of secondarystorage system VM which cycle between Starting & Stop states.

Having this alert on the cloudstack too :

Secondary Storage Vm creation failure in zone [zone-1]. Error details: Unable to allocate capacity on zone [2] due to [null].

on the kvm-host-1-dev:

2025-11-05 18:00:55,598 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-2619124658292654102: Processing:  { Ans: , MgmtId: 93042766621082, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException: Cannot invoke "java.net.URI.getScheme()" because "uri" is null
	at com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:180)
	at com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:234)
	at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:258)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3816)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3395)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2240)
	at com.cloud.agent.Agent.processRequest(Agent.java:813)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

on the mgmt-host-1-dev:

2025-11-05 18:00:55,558 DEBUG [c.c.a.t.Request] (Work-Job-Executor-4:[ctx-e6aabaab, job-7/job-93, ctx-58533c99]) (logid:f27498ef) Seq 2-2619124658292654102: Sending  { Cmd , MgmtId: 93042766621082, via: 2(kvm-host-1-dev), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":"2","name":"s-2-VM","state":"Starting","type":"SecondaryStorageVm","cpus":"1","minSpeed":"500","maxSpeed":"500","minRam":"(512.00 MB) 536870912","maxRam":"(512.00 MB) 536870912","arch":"x86_64","os":"Debian GNU/Linux 12 (64-bit)","platformEmulator":"Debian GNU/Linux 12 (64-bit)","bootArgs":" template=domP type=secstorage host=10.10.0.3 port=8250 name=s-2-VM zone=2 pod=2 guid=s-2-VM workers=5 authorized_key=ZWNkc2Etc2hhMi1uaXN0cDI1NiBBQUFBRTJWalpITmhMWE5vWVRJdGJtbHpkSEF5TlRZQUFBQUlibWx6ZEhBeU5UWUFBQUJCQkdVM2p2Njd5NjBvMXJWTzNqMUxrVEk2eDRydTFzZVNSQ3BQMW1GNmZ0OVNxY0VKZnI0Yk51b3VoZURINlY4Zk5rdVdjNCsyZWxySTVsMWEyNUs2NDg4PSBjbG91ZEBibG9vby1rbm94LWNvcmUtbWdtdC0xLWRldgo= resource=com.cloud.storage.resource.PremiumSecondaryStorageResource instance=SecStorage sslcopy=false role=templateProcessor mtu=1500 eth2ip=10.40.0.11 eth2mask=255.255.240.0 gateway=10.40.0.2 public.network.device=eth2 eth0ip=169.254.202.175 eth0mask=255.255.0.0 eth1ip=10.10.0.20 eth1mask=255.255.240.0 mgmtcidr=10.10.0.0/20 localgw=10.10.0.2 private.network.device=eth1 eth3ip=10.20.0.17 eth3mask=255.255.240.0 storageip=10.20.0.17 storagenetmask=255.255.240.0 storagegateway=10.20.0.2","enableHA":"false","limitCpuUse":"false","enableDynamicallyScaleVm":"false","details":{},"uuid":"6f689280-4203-410e-8496-c2fdaeb2ee41","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"86699c9a-2051-4d65-bfe0-cca899d21e3d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d7c0ff52-f8df-3d44-bd21-a5814655ac1d","name":"primary","id":"2","poolType":"NetworkFilesystem","host":"10.20.0.3","path":"/opt/cloudstack/primary","port":"2049","url":"NetworkFilesystem://10.20.0.3/opt/cloudstack/primary/?ROLE=Primary&STOREUUID=d7c0ff52-f8df-3d44-bd21-a5814655ac1d","isManaged":"false"}},"name":"ROOT-2","size":"(4.88 GB) 5242880000","path":"86699c9a-2051-4d65-bfe0-cca899d21e3d","volumeId":"3","vmName":"s-2-VM","accountId":"1","format":"QCOW2","provisioningType":"THIN","poolId":"2","id":"3","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"0","path":"86699c9a-2051-4d65-bfe0-cca899d21e3d","type":"ROOT","details":{"storageHost":"10.20.0.3","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"(4.88 GB) 5242880000"}}],"nics":[{"deviceId":"2","networkRateMbps":"-1","defaultNic":"true","pxeDisable":"true","nicUuid":"ed48d91d-e234-47a7-832b-6743ef1e8ba5","details":{"MacAddressChanges":"true","MacLearning":"false","ForgedTransmits":"true","PromiscuousMode":"false"},"dpdkEnabled":"false","networkId":"200","networkSegmentName":"D1-A1-Z2-S200","uuid":"e0231d70-d9ef-4e03-b654-1b31728e8472","ip":"10.40.0.11","netmask":"255.255.240.0","gateway":"10.40.0.2","mac":"1e:00:45:00:00:0d","dns1":"1.1.1.1","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","securityGroupEnabled":"false","name":"cloudbr-public"},{"deviceId":"0","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"9313a299-8dd9-46eb-97b4-e972fb84e117","details":{"MacAddressChanges":"true","MacLearning":"false","ForgedTransmits":"true","PromiscuousMode":"false"},"dpdkEnabled":"false","networkId":"202","networkSegmentName":"D1-A1-Z2-S202","uuid":"9f6e38a3-54c3-4ad7-967b-21a4c999bdf2","ip":"169.254.202.175","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:ca:af","broadcastType":"LinkLocal","type":"Control","securityGroupEnabled":"false"},{"deviceId":"1","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"32071063-7f99-4e1e-9d42-e5afe0d8a9d7","details":{"MacAddressChanges":"true","MacLearning":"false","ForgedTransmits":"true","PromiscuousMode":"false"},"dpdkEnabled":"false","networkId":"201","networkSegmentName":"D1-A1-Z2-S201","uuid":"e62966da-f73f-431a-a456-95343e16a9cc","ip":"10.10.0.20","netmask":"255.255.240.0","gateway":"10.10.0.2","mac":"1e:00:52:00:00:0b","broadcastType":"Native","type":"Management","securityGroupEnabled":"false","name":"cloudbr-mgmt"},{"deviceId":"3","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"c14199ec-4cc1-44fe-8076-16034bfbc135","details":{"MacAddressChanges":"true","MacLearning":"false","ForgedTransmits":"true","PromiscuousMode":"false"},"dpdkEnabled":"false","networkId":"203","networkSegmentName":"D1-A1-Z2-S203","uuid":"0da5dec0-a705-4b3c-b717-5a072d44f3bc","ip":"10.20.0.17","netmask":"255.255.240.0","gateway":"10.20.0.2","mac":"1e:00:37:00:00:1e","broadcastType":"Storage","type":"Storage","securityGroupEnabled":"false","name":"cloudbr-storage"}],"vcpuMaxLimit":"1","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{},"metadataManufacturer":"Apache Software Foundation","metadataProductName":"CloudStack KVM Hypervisor"},"hostIp":"10.10.0.6","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.202.175","port":"3922","interval":"6","retries":"100","name":"s-2-VM","wait":"0","bypassHostMaintenance":"false"}}] }
2025-11-05 18:00:55,598 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:[]) (logid:) Seq 2-2619124658292654102: Processing:  { Ans: , MgmtId: 93042766621082, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException: Cannot invoke "java.net.URI.getScheme()" because "uri" is null
	at com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:180)
	at com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:234)
	at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:258)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3816)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3395)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2240)
	at com.cloud.agent.Agent.processRequest(Agent.java:813)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped by previous failure","wait":"0","bypassHostMaintenance":"false"}}] }
2025-11-05 18:00:55,598 DEBUG [c.c.a.t.Request] (Work-Job-Executor-4:[ctx-e6aabaab, job-7/job-93, ctx-58533c99]) (logid:f27498ef) Seq 2-2619124658292654102: Received:  { Ans: , MgmtId: 93042766621082, via: 2(kvm-host-1-dev), Ver: v1, Flags: 10, { Answer, Answer } }
2025-11-05 18:00:55,609 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-4:[ctx-e6aabaab, job-7/job-93, ctx-58533c99]) (logid:f27498ef) Cleaning up resources for the vm VM instance {"id":2,"instanceName":"s-2-VM","state":"Starting","type":"SecondaryStorageVm","uuid":"6f689280-4203-410e-8496-c2fdaeb2ee41"} in Starting state

services are up and running fine: on mgmt cloudstack-management.service and on kvm cloudstack-agent.service

Answered by n4l5u0r

Nov 5, 2025

ok issue solved after manual fix using the approach here : #11557 (comment)

this a regression but occurs also on fresh install not only on upgrades.

View full answer

n4l5u0r · 2025-11-05T19:11:41Z

n4l5u0r
Nov 5, 2025
Author

ok issue solved after manual fix using the approach here : #11557 (comment)

this a regression but occurs also on fresh install not only on upgrades.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Having issues on activating the Advanced Zone on 4.21.0 #11993

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Having issues on activating the Advanced Zone on 4.21.0 #11993

Uh oh!

n4l5u0r Nov 5, 2025

Replies: 1 comment

Uh oh!

n4l5u0r Nov 5, 2025 Author

n4l5u0r
Nov 5, 2025

n4l5u0r
Nov 5, 2025
Author