pure-MPI implementation in a Podman container

[moustakas]

This PR implements a pure-MPI version of FastSpecFit, which does not make use of multiprocessing at all. In addition, a given production (e.g., Y3/Loa) can be run entirely out of a Podman container, which gives us full control over the input software stack (see here for details, including the instructions file).

In production, parallelism is controlled by bin/mpi-fastspecfit. The number of MPI tasks in the mpi4py.MPI.COMM_WORLD communicator (as given by srun --ntasks=n) is split into int(ceil(n/mp)) sub-communicators, where mp is the number of desired ranks per sub-communicator. Using the rank=0 ranks in the sub-communicators, we parallelize over healpixels; in addition, all the ranks in a given sub-communicator are used to parallelize over objects / targets in fastspecfit.fastspec(). In particular, in fastspecfit.fastspec() we use comm.send and comm.recv (send and receive) to make sure each rank only sees the data it needs, which should help prevent memory overflow problems.

I'll post timing tests shortly. And at the moment, there is one issue that I have not been able to track down, but perhaps others may have some ideas.