Fix deadlock on I/O errors during device removal

[amotin]

spa_vdev_remove_thread() should not hold svr_lock while loading metaslabs. It may block ZIO threads, required to handle metaslab loading, at least in case of read errors causing recovery writes.

How Has This Been Tested?

I hope it to fix deadlocks of removal_with_errors ZTS test that for some reason I observed plenty in the CI lately.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[tonyhutter]

Can we get one more reviewer on this one?

[robn]

The approach seems right to me.

The only possible thing I can thing might go wrong is if two threads raced through here such that one loaded a range tree and swapped it into svr_allocd_segs, and then the other did. I suspect that won't happen or won't be a problem for other reasons though, based on normal ZFS patterns and my own intuition:

• there's only one thread?
• the locking dance can't fall out this way
• its only ever going to swap the exact same contents in, so it's an effective no-op

Because I don't know this code at all and have not thought it all the way through, and because I am quite sure you know what you're doing here, take this as "well, I did at least think about it a little bit and its almost certainly fine" :)

Thanks!

[amotin]

@robn Thanks. Yes, AFAIK there can be only one thread. Plus the code asserts svr_allocd_segs is empty, first explicitly, and then (to my surprise) as part of zfs_range_tree_swap(). So it don't think there is a chance of double loading.