Add keys "openmpi" and "mpich" in mpi to specify which MPI implementa…

…tion to

use.
Add mpich hello world test configuration
Add orterun and mpiexec.hydra keys in show key option
Add MPICH and process mapping example in MPI documentation

auto_write_tests.sh

@@ -37,8 +37,10 @@ tee $script_dir/build-lsf-example.txt <<<"buildtest build -c $BUILDTEST_ROOT/too
 tee $script_dir/build-slurm-example.txt <<<"buildtest build -c $BUILDTEST_ROOT/toolkit/buildtest/suite/compilers/helloworld/hello_slurm.yml -vv""" | bash >>$script_dir/build-slurm-example.txt
 
 # MPI example
-tee $script_dir/build-mpi-example1.txt <<<"buildtest build -mc 0 -vv -c $BUILDTEST_ROOT/toolkit/buildtest/suite/mpi/examples/hello.c.yml """ | bash >>$script_dir/build-mpi-example1.txt
-tee $script_dir/build-mpi-example2.txt <<<"buildtest build -mc 0 -vv -c $BUILDTEST_ROOT/toolkit/buildtest/suite/mpi/examples/mpi_ping.c.slurm.yml """ | bash >>$script_dir/build-mpi-example2.txt
+tee $script_dir/build-openmpi-example1.txt <<<"buildtest build -mc 0 -vv -c $BUILDTEST_ROOT/toolkit/buildtest/suite/mpi/examples/hello.c.yml """ | bash >>$script_dir/build-openmpi-example1.txt
+tee $script_dir/build-openmpi-example2.txt <<<"buildtest build -mc 0 -vv -c $BUILDTEST_ROOT/toolkit/buildtest/suite/mpi/matrixmux/mm_mpi.f.yml """ | bash >>$script_dir/build-openmpi-example2.txt
+tee $script_dir/build-mpich-example1.txt <<<"buildtest build -mc 1 -vv -c $BUILDTEST_ROOT/toolkit/buildtest/suite/mpi/examples/hello.c.mpich.yml """ | bash >>$script_dir/build-mpich-example1.txt
+tee $script_dir/build-srun-example1.txt <<<"buildtest build -mc 0 -vv -c $BUILDTEST_ROOT/toolkit/buildtest/suite/mpi/examples/mpi_ping.c.slurm.yml """ | bash >>$script_dir/build-srun-example1.txt
 # List Subcommand
 tee $script_dir/buildtest-list-help.txt <<<"buildtest list --help" | bash >> $script_dir/buildtest-list-help.txt
 tee $script_dir/buildtest-list-software.txt <<< "buildtest list --software" | bash >> $script_dir/buildtest-list-software.txt

docs/build_subcommand/mpi.rst

@@ -2,13 +2,13 @@ Building a MPI Test
 ====================
 
 To build an mpi program use the ``mpi:`` section in test configuration to define
-mpi configuration. The mpi section follows up with the mpi launcher, currently buildtest
-supports (``mpirun``, ``srun``) followed by options for the respective job launchers.
+the start of mpi configuration.
 
 Simple Example
 ---------------
 
-In lines 3-5, we are compiling a MPI hello.c program using the mpirun launcher.
+In lines 3-5, we are compiling a MPI hello.c program using the OpenMPI
+implementation.
 
 .. code-block::
    :linenos:
@@ -17,7 +17,7 @@ In lines 3-5, we are compiling a MPI hello.c program using the mpirun launcher.
     source: hello.c
     compiler: gnu
     mpi:
-      mpirun:
+      openmpi:
        n: '2'
     flags: -O2
     maintainer:
@@ -67,8 +67,49 @@ command.
 
 Next we build the MPI test using the module collection ``0`` using the option ``-mc 0``
 
-.. program-output:: cat scripts/build-mpi-example1.txt
+.. program-output:: cat scripts/build-openmpi-example1.txt
 
+Notice, buildtest is using ``orterun`` as the job launcher when writing the
+test script which is the job launcher for OpenMPI.
+
+Running MPI with MPICH
+-----------------------
+
+If you want to build MPI program using MPICH, then make use of ``mpich`` key
+that hints buildtest to make use of ``mpiexec.hydra`` as the job launcher for
+running MPI jobs. In example below, we have same program (hello.c) that is
+built with MPICH
+
+
+.. code-block::
+   :linenos:
+   :emphasize-lines: 3-5
+
+    source: hello.c
+    compiler: gnu
+    mpi:
+      mpich:
+       n: '2'
+    flags: -O2
+    maintainer:
+    - shahzeb siddiqui [email protected]
+    testblock: singlesource
+
+For this test, we are using ``gmpich/2017.08`` toolchain that consist of
+``MPICH/3.2.1-GCC-7.2.0-2.29`` and its deps built using easybuild. This module
+is added to module collection as index **1**.
+
+::
+
+    $ buildtest module collection -l
+    0: ['GCCcore/8.2.0', 'zlib/1.2.11-GCCcore-8.2.0', 'binutils/2.31.1-GCCcore-8.2.0', 'GCC/8.2.0-2.31.1', 'numactl/2.0.12-GCCcore-8.2.0', 'XZ/5.2.4-GCCcore-8.2.0', 'libxml2/2.9.8-GCCcore-8.2.0', 'libpciaccess/0.14-GCCcore-8.2.0', 'hwloc/1.11.11-GCCcore-8.2.0', 'OpenMPI/3.1.3-GCC-8.2.0-2.31.1', 'OpenBLAS/0.3.5-GCC-8.2.0-2.31.1', 'gompi/2019a', 'FFTW/3.3.8-gompi-2019a', 'ScaLAPACK/2.0.2-gompi-2019a-OpenBLAS-0.3.5', 'foss/2019a']
+    1: ['GCCcore/7.2.0', 'zlib/1.2.11-GCCcore-7.2.0', 'binutils/2.29-GCCcore-7.2.0', 'GCC/7.2.0-2.29', 'MPICH/3.2.1-GCC-7.2.0-2.29', 'gmpich/2017.08']
+
+To build this test use the ``-mc 1`` to load the MPICH module into test script.
+
+.. program-output:: cat scripts/build-mpich-example1.txt
+
+Notice ``mpiexec.hydra`` is being invoked on the executable ``hello.c.exe``.
 
 Running MPI program with srun
 ------------------------------
@@ -104,5 +145,65 @@ the foss toolchain that is needed to build the program mpi_ping.c. When ``slurm:
 is defined in test configuration, buildtest will use the ``.slurm`` extension when writing test
 
 
-.. program-output:: cat scripts/build-mpi-example2.txt
+.. program-output:: cat scripts/build-srun-example1.txt
+
+
+MPI Example with Binding and Process Mapping
+---------------------------------------------
+
+To retrieve bindings for launched process and mapped located of each process,
+then one can use the ``--report-bindings`` and ``--display-map`` option that
+is part of ``orterun``. In buildtest this can be set using the
+``report-bindings`` and ``display-map`` keys. Since this is an optional
+argument, we set value to an empty string.
+
+
+.. code-block::
+   :linenos:
+   :emphasize-lines: 7-8
+
+    source: mpi_mm.f
+    compiler: gnu
+    mpi:
+      openmpi:
+        n: '2'
+        npernode: '2'
+        report-bindings: ''
+        display-map: ''
+    flags: -O2
+
+
+    maintainer:
+    - shahzeb siddiqui [email protected]
+    testblock: singlesource
+
+
+Next we will build this test using collection ``0`` (**foss/2019a**).
+
+.. program-output:: cat scripts/build-openmpi-example2.txt
+
+Notice the options ``--report-bindings`` and ``--display-map`` are added to
+the ``orterun`` command. If we run this test manually we will see the process
+mapping at start of execution::
+
+
+    $ sh /tmp/ec2-user/buildtest/tests/suite/mpi/matrixmux/mm_mpi.f.yml.sh
+     Data for JOB [15602,1] offset 0 Total slots allocated 2
+
+     ========================   JOB MAP   ========================
+
+     Data for node: buildtest       Num slots: 2    Max slots: 0    Num procs: 2
+            Process OMPI jobid: [15602,1] App: 0 Process rank: 0 Bound: socket 0[core 0[hwt 0]]:[B/.]
+            Process OMPI jobid: [15602,1] App: 0 Process rank: 1 Bound: socket 0[core 1[hwt 0]]:[./B]
 
+     =============================================================
+    [buildtest:17659] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/.]
+    [buildtest:17659] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B]
+     task ID=            0
+        sending           7  cols to task           1
+     task ID=            1
+          0.00   1015.00   2030.00   3045.00   4060.00   5075.00   6090.00
+          0.00   1120.00   2240.00   3360.00   4480.00   5600.00   6720.00
+          0.00   1225.00   2450.00   3675.00   4900.00   6125.00   7350.00
+          0.00   1330.00   2660.00   3990.00   5320.00   6650.00   7980.00
+          0.00   1435.00   2870.00   4305.00   5740.00   7175.00   8610.00

docs/scripts/build-mpich-example1.txt

@@ -0,0 +1,29 @@
+buildtest build -mc 1 -vv -c /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/examples/hello.c.mpich.yml 
+________________________________________________________________________________
+compiler: gnu
+flags: -O2
+maintainer:
+- shahzeb siddiqui [email protected]
+mpi:
+  mpich:
+    n: '2'
+source: hello.c
+testblock: singlesource
+
+________________________________________________________________________________
+Key Check PASSED for file /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/examples/hello.c.mpich.yml
+Source File /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/examples/src/hello.c exists!
+Programming Language Detected: c
+Compiler Check Passed
+/tmp/ec2-user/buildtest/tests/suite/mpi/examples
+Writing Test: /tmp/ec2-user/buildtest/tests/suite/mpi/examples/hello.c.mpich.yml.sh
+Changing permission to 755 for test: /tmp/ec2-user/buildtest/tests/suite/mpi/examples/hello.c.mpich.yml.sh
+________________________________________________________________________________
+#!/usr/bin/sh
+module purge 
+module load GCCcore/7.2.0 zlib/1.2.11-GCCcore-7.2.0 binutils/2.29-GCCcore-7.2.0 GCC/7.2.0-2.29 MPICH/3.2.1-GCC-7.2.0-2.29 gmpich/2017.08
+cd /tmp/ec2-user/buildtest/tests/suite/mpi/examples
+mpicc -O2 -o hello.c.exe /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/examples/src/hello.c 
+mpiexec.hydra -n 2 ./hello.c.exe 
+ rm ./hello.c.exe 
+________________________________________________________________________________

docs/scripts/build-mpi-example1.txt → docs/scripts/build-openmpi-example1.txt

@@ -5,7 +5,7 @@ flags: -O2
 maintainer:
 - shahzeb siddiqui [email protected]
 mpi:
-  mpirun:
+  openmpi:
     n: '2'
 source: hello.c
 testblock: singlesource
@@ -24,6 +24,6 @@ module purge
 module load GCCcore/8.2.0 zlib/1.2.11-GCCcore-8.2.0 binutils/2.31.1-GCCcore-8.2.0 GCC/8.2.0-2.31.1 numactl/2.0.12-GCCcore-8.2.0 XZ/5.2.4-GCCcore-8.2.0 libxml2/2.9.8-GCCcore-8.2.0 libpciaccess/0.14-GCCcore-8.2.0 hwloc/1.11.11-GCCcore-8.2.0 OpenMPI/3.1.3-GCC-8.2.0-2.31.1 OpenBLAS/0.3.5-GCC-8.2.0-2.31.1 gompi/2019a FFTW/3.3.8-gompi-2019a ScaLAPACK/2.0.2-gompi-2019a-OpenBLAS-0.3.5 foss/2019a
 cd /tmp/ec2-user/buildtest/tests/suite/mpi/examples
 mpicc -O2 -o hello.c.exe /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/examples/src/hello.c 
-mpirun -n 2 ./hello.c.exe 
+orterun -n 2 ./hello.c.exe 
  rm ./hello.c.exe 
 ________________________________________________________________________________

docs/scripts/build-openmpi-example2.txt

@@ -0,0 +1,32 @@
+buildtest build -mc 0 -vv -c /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/matrixmux/mm_mpi.f.yml 
+________________________________________________________________________________
+compiler: gnu
+flags: -O2
+maintainer:
+- shahzeb siddiqui [email protected]
+mpi:
+  openmpi:
+    display-map: ''
+    n: '2'
+    npernode: '2'
+    report-bindings: ''
+source: mpi_mm.f
+testblock: singlesource
+
+________________________________________________________________________________
+Key Check PASSED for file /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/matrixmux/mm_mpi.f.yml
+Source File /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/matrixmux/src/mpi_mm.f exists!
+Programming Language Detected: fortran
+Compiler Check Passed
+/tmp/ec2-user/buildtest/tests/suite/mpi/matrixmux
+Writing Test: /tmp/ec2-user/buildtest/tests/suite/mpi/matrixmux/mm_mpi.f.yml.sh
+Changing permission to 755 for test: /tmp/ec2-user/buildtest/tests/suite/mpi/matrixmux/mm_mpi.f.yml.sh
+________________________________________________________________________________
+#!/usr/bin/sh
+module purge 
+module load GCCcore/8.2.0 zlib/1.2.11-GCCcore-8.2.0 binutils/2.31.1-GCCcore-8.2.0 GCC/8.2.0-2.31.1 numactl/2.0.12-GCCcore-8.2.0 XZ/5.2.4-GCCcore-8.2.0 libxml2/2.9.8-GCCcore-8.2.0 libpciaccess/0.14-GCCcore-8.2.0 hwloc/1.11.11-GCCcore-8.2.0 OpenMPI/3.1.3-GCC-8.2.0-2.31.1 OpenBLAS/0.3.5-GCC-8.2.0-2.31.1 gompi/2019a FFTW/3.3.8-gompi-2019a ScaLAPACK/2.0.2-gompi-2019a-OpenBLAS-0.3.5 foss/2019a
+cd /tmp/ec2-user/buildtest/tests/suite/mpi/matrixmux
+mpifort -O2 -o mpi_mm.f.exe /home/ec2-user/PycharmProjects/buildtest-framework/toolkit/buildtest/suite/mpi/matrixmux/src/mpi_mm.f 
+orterun -n 2 -npernode 2 --report-bindings  --display-map  ./mpi_mm.f.exe 
+ rm ./mpi_mm.f.exe 
+________________________________________________________________________________

docs/scripts/buildtest-show-key.txt

@@ -35,18 +35,27 @@ nodes                | #SBATCH --nodes                | number of nodes on which
 ntasks               | #SBATCH --ntasks               | number of tasks to run.       
 ntasks-per-node      | #SBATCH --ntasks-per-node      | number of tasks to invoke on each node
 time                 | #SBATCH --time                 | time limit                    
+
                                           MPI Keys
 
 Keys                 | MPI Launchers                  | Description                   
 ----------------------------------------------------------------------------------------------------
-mpirun               | mpirun                         | use the mpirun job launcher   
+mpich                | mpich                          | use the mpiexec.hydra job launcher
+openmpi              | openmpi                        | use the orterun job launcher  
 srun                 | srun                           | use the srun job launcher     
-                                       MPIRUN Keys
 
-Keys                 | MPIRUN Options                 | Description                   
+                                      ORTERUN Keys
+
+Keys                 | ORTERUN Options                | Description                   
 ----------------------------------------------------------------------------------------------------
 display-map          | --display-map                  | Display a table showing the mapped location of each process prior to launch.
 n                    | -n                             | Run  this many copies of the program on the given nodes
 npernode             | -npernode                      | On each node, launch this many processes.
 npersocket           | --npersocket                   | On each node, launch this many processes times the number of processor sockets on the node
 report-bindings      | --report-bindings              | Report any bindings for launched processes.
+
+                                MPIEXEC.HYDRA Keys
+
+Keys                 | MPIEXEC.HYDRA Options          | Description                   
+----------------------------------------------------------------------------------------------------
+n                    | -n                             | Run  this many copies of the program on the given nodes

src/buildtest/tools/buildsystem/singlesource.py

@@ -33,6 +33,7 @@
 
 from buildtest.tools.config import config_opts
 from buildtest.tools.file import create_dir, is_file
+from buildtest.tools.mpi import openmpi_opt_table, mpich_opt_table
 from buildtest.tools.yaml import TEMPLATE_SINGLESOURCE, SUPPORTED_COMPILERS, \
     SUPPORTED_MPI_WRAPPERS, SUPPORTED_MPI_LAUNCHERS, SUPPORTED_MPI_FLAVORS, \
     TEMPLATE_JOB_LSF, TEMPLATE_JOB_SLURM, \
@@ -122,25 +123,6 @@ def _check_slurm(self,slurm_dict):
                       + f"Expecting Type: {str(type(TEMPLATE_JOB_SLURM[k]))}")
                 sys.exit(1)
 
-    def _mpirun_opt_table(self, opts):
-        """Translate mpirun YAML keys to mpirun options"""
-        opt_table = {
-            "n": "-n",
-            "npernode": "-npernode",
-            "npersocket": "-npersocket",
-            "report-bindings": "--report-bindings",
-            "display-devel-map": "--display-devel-map",
-            "display-map": "--display-map",
-        }
-
-        val_list = []
-        # check if mpirun option
-        for opt in opts:
-            if opt in opt_table:
-                val_list.append(opt_table[opt])
-
-        return val_list
-
     def _parse(self):
         """ Parse yaml file to determine if content follows the defined yaml
         schema."""
@@ -242,14 +224,24 @@ def _parse(self):
             #  and add it directly as part of run command
             if "srun" in test_dict["mpi"].keys():
                 testscript_dict["run"].append("srun")
-            elif "mpirun" in test_dict["mpi"].keys():
+            # add openmpi yaml keys to build command orterun command
+            elif "openmpi" in test_dict["mpi"].keys():
 
-                testscript_dict["run"].append("mpirun")
-                mpirun_opts = test_dict["mpi"]["mpirun"].keys()
-                mpirun_flags = self._mpirun_opt_table(mpirun_opts)
+                testscript_dict["run"].append("orterun")
+                mpirun_opts = test_dict["mpi"]["openmpi"].keys()
+                mpirun_flags = openmpi_opt_table(mpirun_opts)
                 for flag,opts in zip(mpirun_flags,mpirun_opts):
                     testscript_dict["run"].append(flag)
-                    testscript_dict["run"].append(test_dict["mpi"]["mpirun"][opts])
+                    testscript_dict["run"].append(test_dict["mpi"]["openmpi"][opts])
+            # add mpich yaml keys to build command mpiexec.hydra
+            elif "mpich" in test_dict["mpi"].keys():
+                testscript_dict["run"].append("mpiexec.hydra")
+                mpirun_keys = test_dict["mpi"]["mpich"].keys()
+                mpirun_flags = mpich_opt_table(mpirun_keys)
+                for flag, opts in zip(mpirun_flags, mpirun_keys):
+                    testscript_dict["run"].append(flag)
+                    testscript_dict["run"].append(
+                        test_dict["mpi"]["mpich"][opts])
 
             testscript_dict["run"].append(f"./{exec_name}")
 

src/buildtest/tools/mpi.py

@@ -0,0 +1,55 @@
+############################################################################
+#
+#  Copyright 2017-2019
+#
+#   https://github.com/HPC-buildtest/buildtest-framework
+#
+#  This file is part of buildtest.
+#
+#    buildtest is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    buildtest is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with buildtest.  If not, see <http://www.gnu.org/licenses/>.
+#############################################################################
+
+
+def openmpi_opt_table(opts):
+    """Translate mpirun YAML keys to mpirun options"""
+    opt_table = {
+        "n": "-n",
+        "npernode": "-npernode",
+        "npersocket": "-npersocket",
+        "report-bindings": "--report-bindings",
+        "display-devel-map": "--display-devel-map",
+        "display-map": "--display-map",
+    }
+
+    val_list = []
+    # check if mpirun option
+    for opt in opts:
+        if opt in opt_table:
+            val_list.append(opt_table[opt])
+
+    return val_list
+
+def mpich_opt_table(opts):
+    """Translate mpirun YAML keys to mpirun options"""
+    opt_table = {
+        "n": "-n",
+    }
+
+    val_list = []
+    # check if mpirun option
+    for opt in opts:
+        if opt in opt_table:
+            val_list.append(opt_table[opt])
+
+    return val_list