switch from async to rayon [v3]

[Stebalien]

This switches from async rust to using rayon for conformance test parallelism. I'm making this PR against FVM v3 because we currently have conformance tests there.

Motivation:

• Primary: Remove the need for maintainers to understand/work with complex async code.
• Secondary: Remove async-std (deprecated).

Performance:

• Startup performance for the conformance tests is significantly slower (something to do with locking when we compile the built-in actors from multiple threads?).
• Runtime performance appears to be the same.

Overall, the conformance tests go from 6 to 12 seconds which isn't great (2x) but that extra time appears to be entirely "startup" cost and shouldn't scale with the number of tests.

fixes #2144

[Stebalien]

@rvagg if this isn't easier to understand, it's not worth it. I thought it was going to be simpler, but error handling with rayon was surprisingly difficult. It might be better to re-try this with channels and a thread-pool, although error handling will still be tricky.

My issue with async-await is that it comes with a bunch of sharp edges around moving things and the error messages can be cryptic.

[rvagg]

I'm not understanding how all the code in this function manage to be get deleted in this refactor, what is rayon giving us that makes all of this go away? Or does the deletion come mainly from basic cleanup in the process?

[Stebalien]

A lot of this code was duplicated from MessageVector::from_file (not sure why). There's also some stuff here that makes it possible to configure test-vector runs with environment variables, but we've never really used them.

[rvagg]

This all makes sense to me and is much simpler, but I'm not seeing the error handling difficulty this introduces, can you highlight that for me because it looks like normal rust shenanigans to my eyes. Is it the need for Counters that's the hassle?

The biggest delta is that looking at the new code I'm having to take for granted that using a MultiEngine with the specified parallelism actually delivers the parallelism we want, whereas with async/await you can see it because it's local. And the performance hit is a concern, but you're verifying somehow that it's all in startup? Is that from logging output that you're seeing that? If we can confirm that it's startup cost then maybe we're not properly memoising multi-threaded compile and that sounds like an issue we can deal separately to this, then I'm happy with this change.

[Stebalien]

This all makes sense to me and is much simpler, but I'm not seeing the error handling difficulty this introduces, can you highlight that for me because it looks like normal rust shenanigans to my eyes. Is it the need for Counters that's the hassle?

It's normal rust shenanigans. My hope was that rayon would have "try" versions of all their operations that could fail the pipeline early, but no such luck.

The other way to do this is to manually spin up a worker pool and feed the workers with channels (go style). But I'm not sure if that'll be any better.

The biggest delta is that looking at the new code I'm having to take for granted that using a MultiEngine with the specified parallelism actually delivers the parallelism we want, whereas with async/await you can see it because it's local.

I'm not sure I understand:

1. Before, we constructed a global multi-engine with a specified parallelism.
2. Now we construct a local multi-engine with the parallelism auto-detected by rayon (usually the number of CPU threads).

And the performance hit is a concern, but you're verifying somehow that it's all in startup? Is that from logging output that you're seeing that?

Yeah, I verified it by logging. We get stuck compiling wasm modules then blow past once we're done.

If we can confirm that it's startup cost then maybe we're not properly memoising multi-threaded compile and that sounds like an issue we can deal separately to this, then I'm happy with this change.

I dug into this more and the issue is that:

1. Wasmtime is also using rayon to parallelized compilation of single modules.
2. We're blocking all of the threads in the rayon thread pool waiting on wasmtime to finish compiling the wasm modules.

This is apparently a well-known rayon issue with no good fix. I've tried some nasty yield hacks but... I only managed to shave off 2 seconds when I should be able to shave off 6.

Let's chat about this in person. I think we may just need to shelve this for now.