Improve resilver behavior of attach/replace

[eharris]

Describe the feature would like to see added to OpenZFS

It seems like a great enhancement to zfs would be to add a new command line switch to attach/replace that informs the resilver that the device is known to have at least some good data already on the device.

This would trigger a scrub-like read-before-write policy for the resilver process on the given device.

How will this feature improve OpenZFS?

Utilizing a read-before-write policy for certain resilver operations would save unnecessary writes (which would extend the life) on write-limited devices like SSD's, as well as potentially speeding up the resilvering process. Since reads are typically faster than writes (even on magnetic media), while this would be adding an extra read when blocks do need to be written, saving the writes on even a relatively small portion of the device could still be a net gain.

Additional context

This could also be a fallback method to improve issues like #16984, or other circumstances where a removed device might be added back to the same pool, even if zfs doesn't recognize it as part of that pool (e.g. if the zfs signature was wiped/damaged, or a mistake accidentally overwrote part of the drive, etc).

[amotin]

My only question is how often do you see this happening to complicate the code for it? For traditional RAIDs read before write makes sense, since it has to be executed on every dirty reboot as the only way to find what has changed. But ZFS does not do it that often. Even if you remove a drive and reinsert it few days later, ZFS will resilver only the changes made during that time. How often do you corrupt all 4 copies of vdev metadata?

[eharris]

@amotin I referenced #16984 in OP, as I had this happen just yesterday in the circumstance of wanting to re-attach a vdev that was zfs split from a mirror pool. The actual amount of data that had changed was probably less than 1MB, on a 4TB SSD.