Perform handle spill IO outside of locked section in SpillFramework

[zpuller]

Addresses #11830

Moves the IO outside of the critical section in the spillable buffer handle spill functions to allow threads interacting with the SpillFramework to manage the spill state of a handle consistently without being blocked on IO. Eg. if thread t1 is in the middle of spilling, and thread t2 wants to check whether this handle is currently spilling, it doesn't need to wait for the spill IO operation to complete in order to check whether the handle it spillable.

[revans2]

Could you please update the docs for all of the public APIs in SpillableHandle and for SpillableHandle itself to make it clear what APIs need to be thread safe and how they are supposed to be protected. For example spill has no indication as to how it should behave when multiple people call it. We can infer that it one wins and the others fail. I want this mostly so it is clear what the contract is for each of these APIs. spillable being a best effort API intended only for quick filtering is fine, but it needs to be documented so if someone tries to use it in a way that requires it to be exact we know that is wrong and it violates the contract. This also will let me reason about the code and who is calling it so that I can better reason about the correctness.

Also could you please explain how error handling/error recovery is intended to happen when spilling? Like if an exception is thrown while we are in the middle of trying to spill something what should happen. Are we supposed to just keep it in the spilling state?

[zpuller]

Could you please update the docs for all of the public APIs in SpillableHandle and for SpillableHandle itself to make it clear what APIs need to be thread safe and how they are supposed to be protected.

Absolutely yeah, I'm just planning to resolve the other outstanding issues in the PR and ensure I have a clear understanding and answer to the other questions, and then will document accordingly.

[zpuller]

I discussed offline with @abellina and he gave me some ideas on how to rework this to not require a separate lock, but to just move the IO component out of the protected section of spill and swap the buffers atomically. I pushed some changes but I'm also converting this back to draft as it's not fully ready for review yet. I still have to address some comments above, plus I am planning to do some manual testing with pulling in the most recent 25.02 branch changes.

[jihoonson]

I'm triggering a build job to see if some test consistently fails for all PRs.

[jihoonson]

build