{crew}/{mirai} interoperability

[wlandau]

crew version 1.3.0 (next release) will be interoperable with mirai in the persistent worker case. Any crew controller with custom compute profile support (releases forthcoming for crew.cluster and crew.aws.batch) will be able to launch workers that mirai can use for other workflows. Reprex for purrr:

controller <- crew::crew_controller_local(workers = 10L, profile = "default")
controller$launch(n = 10L)
system.time(purrr::map(seq_len(10L), purrr::in_parallel(\(x) Sys.sleep(1))))
#>    user  system elapsed 
#>   0.013   0.003   1.685
controller$terminate()

^{Created on 2025-09-12 with reprex v2.1.1}

The above might be enough because autoscaling might not be useful for purrr or tidymodels. If I'm wrong and auto-scaling would be useful, then the pattern might look something like this:

controller <- crew::crew_controller_local(workers = 10L, profile = "default")
controller$autoscale()
system.time(purrr::map(seq_len(10L), purrr::in_parallel(\(x) Sys.sleep(1))))

controller$autoscale() schedules a self-renewing later command that runs auto-scaling (via controller$scale()). The above code currently hangs though. To activate crew's auto-scaling, purrr::map() would need a way to call later::run_now() at intervals on the custom loop that controller$autoscale() registers.

[shikokuchuo]

I see, thanks. That run_now() call is possible even now with how mirai does collection. But we can figure out the right interface for that down the line.

[shikokuchuo]

As we discussed at conf, it may be that all we need is to add a later::run_now() after each iteration of a mirai_map.

[shikokuchuo]

Per our post-conf discussion, we'll postpone this until we find that there's demand for autoscaling more generally, so as to be worth adding the complexity.

[wlandau]

Sounds good!