document function `rebuild_cmdstan`; add detection of need to rebuild gch headers; improve error message

[mitzimorris]

Summary:

When compilation fails because the user needs to rebuild cmdstan, detect this failure and make a better error message.

Add troubleshooting section to the docs, and document function rebuild_cmdstan.

Description:

This is very frustrating for MacOS users, because when Xcode changes, CmdStan needs to be rebuilt.

[WardBrian]

What version of cmdstan are you using and what is the error you’re getting? A couple versions ago we made a change to try to decrease the frequency of errors here but it’s not clear if we succeeded

[mitzimorris]

ask @bob-carpenter

[bob-carpenter]

Here's the output I got. This time I cut and pasted and searched for 'PCH' and realized it did show up as a cmdstanpy warning. Why isn't this an ERROR since nothing can happen with an out of date PCH? Also, why are you asking people to open a bug report given that this is expected behavior for which we have no better solution?

>>> model = csp.CmdStanModel(stan_file='funnel.stan')
22:49:32 - cmdstanpy - INFO - compiling stan file /Users/bcarpenter/temp2/nuts-funnel/funnel.stan to exe file /Users/bcarpenter/temp2/nuts-funnel/funnel
22:49:33 - cmdstanpy - WARNING - CmdStan's precompiled header (PCH) files may need to be rebuilt.Please run cmdstanpy.rebuild_cmdstan().
If the issue persists please open a bug report
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/lib/python3.10/site-packages/cmdstanpy/model.py", line 257, in __init__
    self.compile(force=str(compile).lower() == 'force', _internal=True)
  File "/opt/homebrew/lib/python3.10/site-packages/cmdstanpy/model.py", line 465, in compile
    self._exe_file = compilation.compile_stan_file(
  File "/opt/homebrew/lib/python3.10/site-packages/cmdstanpy/compilation.py", line 481, in compile_stan_file
    raise ValueError(
ValueError: Failed to compile Stan model '/Users/bcarpenter/temp2/nuts-funnel/funnel.stan'. Console:

--- Translating Stan model to C++ code ---
bin/stanc --filename-in-msg=funnel.stan --o=/Users/bcarpenter/temp2/nuts-funnel/funnel.hpp /Users/bcarpenter/temp2/nuts-funnel/funnel.stan

--- Compiling C++ code ---
clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2020.3/include    -O3 -I src -I stan/src -I stan/lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.4.0 -I stan/lib/stan_math/lib/boost_1.81.0 -I stan/lib/stan_math/lib/sundials_6.1.1/include -I stan/lib/stan_math/lib/sundials_6.1.1/src/sundials    -DBOOST_DISABLE_ASSERTS          -c -include-pch stan/src/stan/model/model_header_15_0.hpp.gch -x c++ -o /Users/bcarpenter/temp2/nuts-funnel/funnel.o /Users/bcarpenter/temp2/nuts-funnel/funnel.hpp
fatal error: file '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/pointer_to_unary_function.h' has been modified since the precompiled header 'stan/src/stan/model/model_header_15_0.hpp.gch' was built: mtime changed (was 1707552691, now 1713018384)
note: please rebuild precompiled header 'stan/src/stan/model/model_header_15_0.hpp.gch'
1 error generated.
make: *** [/Users/bcarpenter/temp2/nuts-funnel/funnel.o] Error 1
rm /Users/bcarpenter/temp2/nuts-funnel/funnel.hpp

Command ['make', 'STANCFLAGS+=--filename-in-msg=funnel.stan', '/Users/bcarpenter/temp2/nuts-funnel/funnel']
	error during processing No such file or directory

Since the meaningful error message had scrolled off my screen:

22:49:33 - cmdstanpy - WARNING - CmdStan's precompiled header (PCH) files may need to be rebuilt.Please run cmdstanpy.rebuild_cmdstan().

When Mitzi and I were looking at the output, we only saw this:

note: please rebuild precompiled header 'stan/src/stan/model/model_header_15_0.hpp.gch'

Then we couldn't figure out which command to call to do that from Python. There's no doc I can find for rebuild_cmdstan() in the CmdStanPy docs using Google search or the search within CmdStanPy docs.

[WardBrian]

Also, why are you asking people to open a bug report given that this is expected behavior for which we have no better solution?

If rebuild_cmdstan() does not resolve the issue, then it is likely that either:

• something broke in cmdstan's support for precompiled headers
• the message is being shown erroneously and something else entirely is preventing compilation

The second is the bigger concern, which is also why we paste the full output now; in the past, if we thought the PCH was the issue, we would show only that warning, but we then had several issues where the error was actually something else, but just mentioned the pch file, so we misdiagnosed it in the code.

Unfortunately, different versions of clang also feature very different error messages for this, so we can't really detect it with 100% accuracy. This is also why it is a warning rather than an error, because it might be wrong.

---

Thanks for the error message. It seems like at some point apple clang started being pickier about the precompiled header even within the same compiler version. I think we can improve this by telling the cmdstan build system to re-build that file when system headers change - before we were explicitly only depending on our own files, but that doesn't work for this kind of artifact really. I added the change to a PR I already have on the PCH feature: stan-dev/cmdstan@c1d0cae

[bob-carpenter]

rebuild_cmdstan() worked, it just took me and Mitzi over 5 minutes to find the name of the function. We didn't catch it buried in the top of the output and only saw the injunction at the end to rebuild.