Figure out if and how to use io_uring inter-ring messages for polling interruption

[armanbilge]

An important feature of the polling system is the ability to interrupt a poller while it is polling. This is necessary to wake up a worker thread e.g. when there is work available in the external queue.

cats-effect/core/jvm-native/src/main/scala/cats/effect/unsafe/PollingSystem.scala

Line 56 in 58f695f

def interrupt(targetThread: Thread, targetPoller: Poller): Unit

io_uring now has the ability to directly send messages between rings. This seems like a great way to implement this interruption, since sending a message to a ring would cause it to wake up immediately.

https://man7.org/linux/man-pages/man3/io_uring_prep_msg_ring.3.html

However, sending said message requires having access to a ring to send the message from. This opens a few questions:

1. should the PollingSystem#interrupt API be given access to the poller of its current thread, if available?

2. if we are not on a WSTP thread and we don't have a thread-local ring, what should we do? This is a common case, since work for the external queue will originate from external threads.

3. even if we do have a thread-local ring we can use to send the message, should we bypass the usual submission mechanism? otherwise it may take up to 64 iterations of the worker loop until the interrupt even gets submitted, which is a lot of latency. seems kinda dumb

So, tough questions. We could implement this today, without access to the thread-local ring, by keeping a separate collection of rings used only for sending interrupt messages. This is is necessary anyway to support sending interrupts from external threads.

Follow-up to:

• Polling system #3332

cc @antoniojimeneznieto

[antoniojimeneznieto]

2. I may be missing something but couldn't we use the separate pool of reserved rings to send interrupt messages ? 🤔 If that's the case, 1. do we need to give access to the current thread poller to the interrupt API?

3. Yes I completely agree, I think we could work on some priority mechanism to avoid waiting for the current iteration to finish.

[armanbilge]

couldn't we use the separate pool of reserved rings to send interrupt messages

yes, you are exactly right. we could always use this pool of rings for interruption, even if we have a local ring. then we would not have to figure out a priority mechanism for the local ring, since we can submit the pool ring immediately.

... but using these pool rings would require some kind of synchronization / locking overhead, which is not needed with a local ring.

[antoniojimeneznieto]

... but using these pool rings would require some kind of synchronization / locking overhead, which is not needed with a local ring.

Right, I see. I think it's a good idea to use poller from the current thread to avoid synchronization. But we run into the problem of 2. A naive approach would be to use the ring pool to deal with these cases but we would be in the same situation again... Maybe we can do something better 🤔 I'll do some research on this.

For (3) I don't think the priority mechanism would be a big problem to develop.