Refactor "standard library" as a virtual module

[WardBrian]

This is an alternative to #1182.

---

This idea came from a discussion with @mandel and @gbdrt at ProbProg 2021. They had been working on an effort to translate Stan programs to Pyro and NumPyro at https://github.com/deepppl, and I asked them what about the current implementation of stanc made it easier or harder to do this kind of work. One thing that came up was the reliance on the Stan math library signatures.

Current implementation

At the moment, we have a file src/middle/Stan_math_signatures.ml which defines a lookup table of all the C++ functions we'd like to expose at the language level. We've essentially been treating these as built in to Stan, but I'd argue this is actually very backend specific. The fact that all these functions need to be defined for your backend to work is part of what stalled the Tensorflow backend, and it created a considerable effort for the deepppl folks, who defined a name-matching Python library: https://github.com/deepppl/stan-num-pyro/blob/main/stan-pyro/stanpyro/stanlib.py. For functions which don't exist or they don't support, they defined dummy signatures which throw a runtime exception.

If we're serious about stanc being architected to support alternative backends, doing something better here is pretty key. I would argue (and I think so would the name of the file) that, as a starting point, Stan_math_signatures belongs in the src/stan_math_backend folder, not the middle folder.

The changes

This uses dune's virtual module system (doc, a great blog post explaining them) to make the "Library" interface abstract over any particular implementation. This means we can treat "Library" like any other module, (e.g., no need for functors or anything like that), and just use dune to link this to a specific implementation.

This means another usage of the compiler producing a different executable could link a different implementation of the same interface. The downside over functors is you cannot create multiple instances per executable, but for the compiler usecase I can't imagine you would want that anyway.

This has the same primary benefit as #1182, but without the downsides of the functors. Most of the changes in this smaller version of the diff are simply cleanup/refactors which make the Library abstraction work and (imo) are good to have even without the added modularity this gains.

---

Submission Checklist

• Run unit tests
• Documentation
  • OR, no user-facing changes were made

Release notes

Refactored internals of the compiler to be less tied to the specific functions exposed in the Stan Math C++ library.

Copyright and Licensing

By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the BSD 3-clause license (https://opensource.org/licenses/BSD-3-Clause)

[SteveBronder]

Had a look at about half of this today and skimmed the other half. I think this is good and I like it, but there is a lot of code so it's going to take me a bit of time to review. But in general I like how things are broken up here (especially the math signatures being moved to the stan math backend, pulling out the variadic stuff, etc.).

[WardBrian]

Thanks @SteveBronder - it's a lot of moving parts to keep track of but I agree I quite like how it ended up looking

[WardBrian]

During this months language meeting @mandel pointed out some use-cases for this refactor which this version doesn't serve very well. I hadn't considered that we may want more than one library per executable, but it turns out this is how deepppl is implemented, and @bob-carpenter also pointed out the potential for us wanting to be able to restrict the stan math library even more (e.g., to only functions which support reverse mode).

I've marked this as a draft and I'm planning to look at something like #1182 again with fresh eyes

[WardBrian]

Note: Virtual modules also do not work with dune utop, which is unfortunate for debugging.

[WardBrian]

As time goes on I think more and more that #1182 is the right way to go on this, so I'm going to close this version