Minimal and Generic Cmdstanpy for Deployment

[winedarksea]

I am working in a restricted, proprietary code environment. I have run into a wall trying to import the full cmdstan stack, a mess of Python, C, and Stan, all expecting things located in certain directories (which defaults won't be usable here).

My goal is just to use a single stan model, like prophet_model.stan from the Prophet forecasting package. So my question is, can I:

1. Build the stan file into a model.bin on another computer (where I've got everything working from a conda-forge install, not available in prod)
2. Transfer the model.bin to a new environment (similar x86 64 bit but not exactly the same processor, perhaps a different linux distro but reasonably up to date Linux on both)
3. Install cmdstanpy in new environment but not having to compile or install anything else in C, just pure python
4. Use cmdstanpy to load the model.bin and fit it, without any other dependencies

And if yes to those things, is there a way to enable compiler flags for the stan model for just something like AVX2 but otherwise be as generic as possible?

thanks in advance!

[WardBrian]

The short answer is "yes". Prophet's python package essentially does exactly what you describe, just inside a Python wheel.

Now, whether or not your sysadmin will be happier with a precompiled executable from somewhere, I can't say

Let's call the build machine "B" and the prod machine "P". Here are the rough steps you would need to take

• Build CmdStan on B. Use it to build model.bin.
• Copy a few files from B to P. These files are model.bin, $CMDSTAN/stan/lib/stan_math/lib/tbb/libtbb.so, and $CMDSTAN/stan/lib/stan_math/lib/tbb/libtbb.so.2. The easiest thing to do is probably place all 3 of these files in the same folder on P.
  • The second and third files are the only part of this that is "tricky". Stan needs the Intel Threaded Building Blocks shared library at runtime.
  • You may run into issues with model.bin failing to find tbb on P, if so there may need to be another step performed on B using something like patchelf
  • If you have a specific version of TBB available on machine P, it is probably easier to install the same version on machine B and convince Stan to use it for building, at which point model.bin is all you would need to copy.
• Initialize the CmdStanModel object with a full path to model.bin, use as normal

There shouldn't be a need to set up a fake cmdstan path inside cmdstanpy if this is all running on Linux. If you have any choice in the manner, you should try to have B be running a strictly older version of glibc to avoid linking issues once you copy it over. You can enable some machine specific optimizations by adding a flag like -march=haswell to the compilation step on B. Haswell is probably a reasonable guess, but you'd want to select the oldest of the two (B and P). The available options are here

I have a template repository at https://github.com/WardBrian/cookiecutter-cmdstanpy-wrapper which walks through the basics of building a prophet-like package wheel, but it assumes 1. you care about macos and windows, and 2. the build machine B is a github actions runner, so it may not all be immediately translatable to your use case.

[winedarksea]

Here's how I ended up doing it with Prophet as an example:

Importing this was a bit of a challenge...

for Prophet == 1.1.5

stanio=0.3.0. (remove numpy typing)
cmdstanpy=1.2.1
prophet (remove numpy typing due to older numpy version still used in places)
(add tbb library LD_LIBRARY_PATH in _load_stan_backend in forecaster.py)
like this:

    def _load_stan_backend(self, stan_backend):
        dir_path = importlib_resources.files("prophet") / "stan_model"
        if "LD_LIBRARY_PATH" in os.environ:
            if "stan_model" not in str(os.environ["LD_LIBRARY_PATH"]):
                os.environ['LD_LIBRARY_PATH'] = dir_path.__str__() + ':' + os.environ['LD_LIBRARY_PATH']
        else:
            os.environ['LD_LIBRARY_PATH'] = dir_path.__str__()

THIS WILL ONLY WORK ON the architecture/os that it was built on

Installing prophet/cmdstanpy/stan from conda-forge is easy, so use that as a starting point, as reference.

I zipped a /bin/cmdstan directory up (full git clone with submodule), moved to the ondemand, unzipped. You may need to link the /oneTBBxx/include/tbb to the ./lib of cmdstan/stan/lib/stan_math something like ln -s /home/colincatlin/tbb/include/tbb ~/cmdstan/stan/lib/stan_math/ but I didn't the last time.

Do the git clone of the cmdstan repo so the submodules are present git clone https://github.com/stan-dev/cmdstan.git --recursive, then zip to send over. Then make build in the cmdstan repo. I also had to manually download linux-stanc from github and move that to cmdstan/bin.

stan model can be compiled something like this:

import cmdstanpy
cmdstanpy.set_cmdstan_path("/home/colincatlin/cmdstan")

cpp_options = {
    "march":"haswell",
    # "I": "/home/colincatlin/tbb/include", "L": "/home/colincatlin/fbsource/third-party/pypi/prophet/1.1.5/prophet/stan_model/",
    # "L": "/home/colincatlin/fbsource/third-party/pypi/prophet/1.1.5/prophet/stan_model",  # Add the path to the TBB library
    "L": "/home/colincatlin/cmdstan/stan/lib/stan_math/lib/tbb",  # Add the path to the TBB library
    # "ltbb": "",  # Link with TBB library
    # "lstdc++": "",
    # "std": "c++1y", "D_REENTRANT": "", "Wno-ignored-attributes": "", "Wno-sign-compare": "",
    # "CXXFLAGS": "-I/home/colincatlin/tbb/include -L/home/colincatlin/fbsource/third-party/pypi/prophet/1.1.5/prophet/stan_model -ltbb",
}
stanc_options = {
    "O": "",  # this optimizer might cause troubles, it is technically experimental, but worked so far
    # "include_paths": f"/home/colincatlin/tbb/include"
}
sm = cmdstanpy.CmdStanModel(stan_file="/home/colincatlin/fbsource/third-party/pypi/prophet/1.1.5/stan/prophet.stan", cpp_options=cpp_options, stanc_options=stanc_options, force_compile=True)
print(sm.exe_file)

move prophet binary object to ...prophet/stan_model/prophet_model.bin
and move the tbb files from /home/colincatlin/cmdstan/stan/lib/stan_math/lib/tbb to the same directory.

[bob-carpenter]

Thanks for reporting back what worked. The "mess" is even worse in that the full stack also requires OCaml to compile new models (luckily you don't need that) and make. And the interfaces not only need C, but also C wrappers for our underlying C++ code.

I'm always amazed by the level of support @WardBrian can provide in a small fraction of his total effort.