Enzyme should be loaded on-demand at runtime

[workingjubilee]

This is an enhancement request for the sake of making the Enzyme autodiff functionality more maintainable: load Enzyme at runtime, on demand, via a mechanism like dlopen. My understanding is that both T-infra (or at least @Kobzol) and the relevant autodiff lead (@ZuseZ4) are interested in seeing this happen, because it would help decouple the Enzyme bindings and thus the infrastructure maintenance required to ship Enzyme support.

I do not fully understand what Enzyme actually "is" (is it actually a shared object that is loaded by LLVM? or do you open it and then it runs LLVM? or do you just have both in the same address space and then... magic happens?) so I cannot describe further what is required other than that. I am opening this issue so we can better identify and track

• what we hope to gain by doing this work?
• how will this affect the API's library support?
• how will this affect the internal compiler support?
• how do we actually do it?
  • we first have to identify how Enzyme and LLVM actually... interact on the procedural level and what role rustc plays
  • @bjorn3 says a complication is from compiling Enzyme as a .so, because LLVM is sometimes "statically"¹ linked by rustc
  • once that is done, I suspect the easiest route is to not use dlopen directly but rather libloading

@rustbot label: +C-enhancement +F-autodiff +A-LLVM +T-compiler +T-infra +T-libs

Footnotes

1. "statically" in this case means "required dynamic linkage", so it is loaded by the dynamic loader upon loading librustc_driver.so ↩