Fix deadlock on I/O errors during device removal #17145

amotin · 2025-03-14T00:10:24Z

spa_vdev_remove_thread() should not hold svr_lock while loading metaslabs. It may block ZIO threads, required to handle metaslab loading, at least in case of read errors causing recovery writes.

How Has This Been Tested?

I hope it to fix deadlocks of removal_with_errors ZTS test that for some reason I observed plenty in the CI lately.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

spa_vdev_remove_thread() should not hold svr_lock while loading a metaslab. It may block ZIO threads, required to handle metaslab loading, at least in case of read errors causing recovery writes. Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc.

tonyhutter · 2025-03-17T17:58:31Z

Can we get one more reviewer on this one?

robn

The approach seems right to me.

The only possible thing I can thing might go wrong is if two threads raced through here such that one loaded a range tree and swapped it into svr_allocd_segs, and then the other did. I suspect that won't happen or won't be a problem for other reasons though, based on normal ZFS patterns and my own intuition:

there's only one thread?
the locking dance can't fall out this way
its only ever going to swap the exact same contents in, so it's an effective no-op

Because I don't know this code at all and have not thought it all the way through, and because I am quite sure you know what you're doing here, take this as "well, I did at least think about it a little bit and its almost certainly fine" :)

Thanks!

amotin · 2025-03-19T15:00:29Z

@robn Thanks. Yes, AFAIK there can be only one thread. Plus the code asserts svr_allocd_segs is empty, first explicitly, and then (to my surprise) as part of zfs_range_tree_swap(). So it don't think there is a chance of double loading.

FreeBSD kernel's WITNESS code detected lock ordering violation in spa_vdev_remove_cancel_sync(). It took svr_lock while holding ms_lock, which is opposite to other places. I was thinking to resolve it similar to openzfs#17145, but looking closer I don't think we even need svr_lock at that point, since we already asserted svr_allocd_segs is empty, and we don't need to add there segments we are going to call free_mapped_segment_cb for. Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc.

FreeBSD kernel's WITNESS code detected lock ordering violation in spa_vdev_remove_cancel_sync(). It took svr_lock while holding ms_lock, which is opposite to other places. I was thinking to resolve it similar to #17145, but looking closer I don't think we even need svr_lock at that point, since we already asserted svr_allocd_segs is empty, and we don't need to add there segments we are going to call free_mapped_segment_cb for. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Allan Jude <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes #17164

spa_vdev_remove_thread() should not hold svr_lock while loading a metaslab. It may block ZIO threads, required to handle metaslab loading, at least in case of read errors causing recovery writes. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17145

FreeBSD kernel's WITNESS code detected lock ordering violation in spa_vdev_remove_cancel_sync(). It took svr_lock while holding ms_lock, which is opposite to other places. I was thinking to resolve it similar to openzfs#17145, but looking closer I don't think we even need svr_lock at that point, since we already asserted svr_allocd_segs is empty, and we don't need to add there segments we are going to call free_mapped_segment_cb for. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Allan Jude <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17164 (cherry picked from commit 301da59)

FreeBSD kernel's WITNESS code detected lock ordering violation in spa_vdev_remove_cancel_sync(). It took svr_lock while holding ms_lock, which is opposite to other places. I was thinking to resolve it similar to openzfs#17145, but looking closer I don't think we even need svr_lock at that point, since we already asserted svr_allocd_segs is empty, and we don't need to add there segments we are going to call free_mapped_segment_cb for. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Allan Jude <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17164

spa_vdev_remove_thread() should not hold svr_lock while loading a metaslab. It may block ZIO threads, required to handle metaslab loading, at least in case of read errors causing recovery writes. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17145

FreeBSD kernel's WITNESS code detected lock ordering violation in spa_vdev_remove_cancel_sync(). It took svr_lock while holding ms_lock, which is opposite to other places. I was thinking to resolve it similar to openzfs#17145, but looking closer I don't think we even need svr_lock at that point, since we already asserted svr_allocd_segs is empty, and we don't need to add there segments we are going to call free_mapped_segment_cb for. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Allan Jude <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17164

spa_vdev_remove_thread() should not hold svr_lock while loading a metaslab. It may block ZIO threads, required to handle metaslab loading, at least in case of read errors causing recovery writes. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17145

FreeBSD kernel's WITNESS code detected lock ordering violation in spa_vdev_remove_cancel_sync(). It took svr_lock while holding ms_lock, which is opposite to other places. I was thinking to resolve it similar to openzfs#17145, but looking closer I don't think we even need svr_lock at that point, since we already asserted svr_allocd_segs is empty, and we don't need to add there segments we are going to call free_mapped_segment_cb for. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Allan Jude <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17164

spa_vdev_remove_thread() should not hold svr_lock while loading a metaslab. It may block ZIO threads, required to handle metaslab loading, at least in case of read errors causing recovery writes. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Rob Norris <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17145

FreeBSD kernel's WITNESS code detected lock ordering violation in spa_vdev_remove_cancel_sync(). It took svr_lock while holding ms_lock, which is opposite to other places. I was thinking to resolve it similar to openzfs#17145, but looking closer I don't think we even need svr_lock at that point, since we already asserted svr_allocd_segs is empty, and we don't need to add there segments we are going to call free_mapped_segment_cb for. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: Allan Jude <[email protected]> Signed-off-by: Alexander Motin <[email protected]> Sponsored by: iXsystems, Inc. Closes openzfs#17164

amotin added the Status: Code Review Needed Ready for review and testing label Mar 14, 2025

amotin mentioned this pull request Mar 17, 2025

kstat: allow multi-level module names #17142

Closed

13 tasks

tonyhutter approved these changes Mar 17, 2025

View reviewed changes

robn approved these changes Mar 19, 2025

View reviewed changes

amotin added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Mar 19, 2025

amotin merged commit 676b7ef into openzfs:master Mar 19, 2025
76 of 88 checks passed

amotin deleted the removal_deadlock branch March 19, 2025 18:48

amotin mentioned this pull request Mar 21, 2025

Fix lock reversal on device removal cancel #17164

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix deadlock on I/O errors during device removal #17145

Fix deadlock on I/O errors during device removal #17145

Uh oh!

amotin commented Mar 14, 2025 •

edited

Loading

Uh oh!

tonyhutter commented Mar 17, 2025

Uh oh!

robn left a comment

Uh oh!

amotin commented Mar 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Fix deadlock on I/O errors during device removal #17145

Fix deadlock on I/O errors during device removal #17145

Uh oh!

Conversation

amotin commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How Has This Been Tested?

Types of changes

Checklist:

Uh oh!

tonyhutter commented Mar 17, 2025

Uh oh!

robn left a comment

Choose a reason for hiding this comment

Uh oh!

amotin commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amotin commented Mar 14, 2025 •

edited

Loading

amotin commented Mar 19, 2025 •

edited

Loading