Correct Thread Exiting (for exit()/exec() etc.)

[rennergade]

I'm realizing this is a bug that we never addressed in NaCl-Lind which is incorrect, we should do this correctly here. The spec for exec says that all threads should be terminated on exec: https://stackoverflow.com/questions/36588387/what-happens-to-threads-when-exec-is-called

This issue is closely related to the issues with exit() we've been talking about here: Lind-Project/RawPOSIX#50

[qianxichen233]

Yeah, this is also an existing bug on exit syscall in lind-wasm that threads aren not terminated when when the main process exit. I wonder what are reliable ways to forcely terminate a running thread? Like pthread_cancel?

[rennergade]

We were handling this using pthread_cancel in lind-nacl, but it was always sort of a workaround since cancelling threads is pretty scary.

I talk a bit about a potential solution here: Lind-Project/RawPOSIX#50 (comment)

[rennergade]

Discussed this today with Alice and Qianxi, writing our thoughts down here as a proposal:

When reaching a state where all threads in a cage have to exit, they can be in two situations: Untrusted userspace, or trusted space

@qianxichen233 has added epoch_interruption for threads, which should allow us to control the behavior of running threads. This should allow us to stop threads that are stuck in userpace.

For example, if a thread calls exit()

• untrusted threads should have their epoch deadline set so they return to wasmtime and can be thread exited.
• trusted threads should just be able to return, but be caught before they return to userspace and be thread exited. There are also situations where a thread could be stuck in a system call, which we'll need to handle but is a solvable corner case for a separate issue (via timeouts on sockets/futexes etc).
• the final thread needs to call the actual exit() syscall from RawPOSIX (or whatever is setup here via grates, there's probably more here for integrating this with 3i)

Things that need to be added to support this:

• We need a flag that is updated for whether a thread is in trusted or untrusted code (should be set at the libc/wasmtime trampoline boundary?)
• Need to keep track of how many threads are in a cage, and decrement it as threads exit
• Need to store epoch handlers in a way that they can be triggered between cages
• Most likely need a flag in RawPOSIX cage struct indicating if the cage is exiting (for above blocking issue)
• most likely need to keep track of if a thread is the main thread (thread launched via fork etc) or a child thread, this will definitely be needed for signals but also need to figure out if the actual exit() is called via only the main thread in Linux, or for something like something exec()

Because we need to store the epoch handlers, it probably makes sense to store some of thead struct in each wasm Instance containing some of the above information.

[JustinCappos]

It's plausible that harshexit could / should be called here. We'll need to discuss. If the memory, etc. the process is likely to be unavailable at this point, this is more appropriate than exit.