KVM hosts got rebooted while adding NFS primary storage

[GerorgeEG]

problem

Multiple hosts got rebooted while adding NFS primary storage
ACS version: 4.19.1.2
KVM:REHL 8.10
NFS : v3 with nolock option
Below is the error from one of host
message log:
java[4095]: WARN [kvm.storage.LibvirtStoragePool] (Thread-1:) (logid:) Process [2479886] for command [/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i -p /virt/NFS path -m /mnt/a288c84e-d100-334b-9bc3-0d79ffe9a610 -h ] encountered the error: [Failed to create /mnt/a288c84e-d100-334b-9bc3-0d79ffe9a610/KVMHA//].
agent log :
WARN [kvm.resource.KVMHAMonitor] (Thread-1:null) (logid:) Write heartbeat for pool [a288c84e-d100-334b-9bc3-0d79ffe9a610] failed: Failed to create /mnt/a288c84e-d100-334b-9bc3-0d79ffe9a610/KVMHA//hb-; try: 5 of 5.

versions

The versions of ACS, hypervisors, storage, network etc..

The steps to reproduce the bug

...

What to do about it?

No response

[DaanHoogland]

@GerorgeEG , did you try mounting the new storage "by hand”?

side note: " /mnt/a288c84e-d100-334b-9bc3-0d79ffe9a610/KVMHA//hb-“ does not look like a valid heartbeat file to try and create. An ip/hostname is missing:

[root@pr10710-t14187-kvm-ol9-kvm1 ~]# ls /mnt/0885b0d3-0a1f-37c3-9244-152d61dd1415/KVMHA/hb-*
/mnt/0885b0d3-0a1f-37c3-9244-152d61dd1415/KVMHA/hb-10.0.33.212  /mnt/0885b0d3-0a1f-37c3-9244-152d61dd1415/KVMHA/hb-10.0.33.218

This might indicate a network configuration mistake.
It is not the root of your problem , but more like another symptom.

Is the directory /mnt/0885b0d3-0a1f-37c3-9244-152d61dd1415 present on your system?
and does it contain ./KVMHA/?

[GerorgeEG]

Hi @DaanHoogland, thanks for picking this up, we found that it was the issue with NFS share and that why mounting was stuck and hosts got rebooted, but we want to avoid reboot of hosts if only one of the Primary Storage is having issue but others are accessible. Is there any way we can prevent the reboot of KVMs. I found one setting and it is already disabled other than I am not able to avoid this issue

Kvm ha fence on storage heartbeat failure (kvm.ha.fence.on.storage.heartbeat.failure)
Proceed fencing the host even the heartbeat failed for only one storage pool

[weizhouapache]

@GerorgeEG

If you want the host not to be rebooted when write heartbeat fails, please add/change the value in agent.properties

reboot.host.and.alert.management.on.heartbeat.timeout=false

and restart cloudstack-agent service

[GerorgeEG]

thanks , will check and validate this in our environment.

[rohityadavcloud]

I hit the same issue few weeks ago and can confirm the agent.properties certainly helped.