Volume failed to attach due to being deleted

[veenadong]

Mayastor 2.5.1: pods are in the Init state with: volume is being deleted error.

Republish the volume resulted in MultiAttach error.

Attaching logs from setup 1:
mayastor-2024-09-03--15-57-59-UTC_unable_publish.tar.gz
mayastor-2024-09-03--19-23-29-UTC_unable_publish.tar.gz

Logs from setup 2:
mayastor-2024-09-06--01-47-16-UTC.tar.gz
In this setup, volume af2940b4-61fa-480e-9d5d-38651e1d01ce is in this state.

[tiagolobocastro]

Please delete the io-engine pod for node glcr-hst01-sc3.cloud1.gl-hpe.net
Seems something is holding the nexus lock, which prevents its deletion

Then allow for some minutes and please take another bundle, thank you.

[veenadong]

We restarted all the io-engine pods, and the setup managed to recover.

1. How do we identify this problem? ie. any key words in the logs?
2. What are the safe steps to restart the io-engine? (To prevent/reduce corruption to the filesystem/disk)

[tiagolobocastro]

In this case it was these kind of logs:

 �[2m2024-09-03T19:23:45.388139Z�[0m �[31mERROR�[0m �[1;31mcore::volume::service�[0m�[31m: �[1;31merror�[0m�[31m: gRPC request 'destroy_nexus' for 'Nexus' failed with 'status: DeadlineExceeded, message: "Failed to acquire access to object within given timeout", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 03 Sep 2024 19:23:44 GMT", "content-length": "0"} }'�[0m

Although there was no data-plane logs for that time period, so not sure what was actually holding up the resource...

To safely restart the io-engine, make sure you don't delete with force but instead allow it to gracefully terminate.
In some cases you might want to drain the targets to another node using the plugin, ex: kubectl-mayastor drain node xxx