deal with pymultinest with Cray MPI at NERSC

[heather999]

See LSSTDESC/desc-help#66 (comment)
The td_env conda environment includes mpi4py and the Cray mpi libraries. Until pymultinest offers a way to set an env variable to skip the attempted import of mpi4py when running on a Cori login node (which includes jupyterLab), we need another way to avoid the mpi4py import.
Contacted the mpi4py developer who suggested:

import sys
sys.modules["mpi4py"] = None

Could use PYTHONSTARTUP in the setup script when $SLURM_JOB_ID is unset (meaning we're on a login node rather than compute node) Perhaps not the best option for all cases:

• starting a batch job from a jupyter notebook, I don't think the python environment would be automatically re-initialized on the compute node, rather the current environment without the mpi4py library available would be inherited.
• setting up the environment on a login node before submitting a batch job which inherits the current environment. This could be avoided if the env is set up directly from the batch job or when attached to the "interactive" queue. Actually, this is fixed by checking if SLURM_JOB_ID is set in the PYTHONSTARTUP script itself
• PYTHONSTARTUP is only utilized for an interactive session, so it won't fix the case where a python script is run on a login node
  So not a perfect solution - but I think this might be ok for now where most users are likely not so concerned about MPI.

[heather999]

Ended up using sitecustomize.py instead and that seems to cover all the use cases properly.

[heather999]

Done!