Spack Working Group @ SDSC - Meeting - 2023/04/21

[mkandes]

Attendance

• Marty Kandes
• Mahidahr Tatineni
• Subhashini Sivagnanam
• Jerry Greenberg
• Nicole Wolter
• Yuwu Chen
• Trevor Cooper

Agenda

1. Upcoming Planned Maintenance on Expanse

Expected date of maintenance: May 8th, 2023

• Update operating system from Rocky 8.5 to Rocky 8.7
• Update Infiniband driver from Mellanox OFED 4.9-4.1.7 to Mellanox OFED 5.8-1.1.2
• Update Lustre client from lfs 2.12.8_6_g5457c37 to lfs 2.15.2
• Update NVIDIA GPU driver from NVIDIA 510.39.01 CUDA 11.6 to NVIDIA 515.65.01 CUDA 11.7

No changes or updates here.

2. Initial deployment of expanse/0.17.3/cpu/b on updated nodes is complete

Moving on to the deployment of expanse/0.17.3/gpu/b this weekend and will then begin performance testing.

3. Update on pre-production testing of expanse/0.17.3/cpu/a and expanse/0.17.3/gpu/a

Updates from @nwolter. Review in-depth next week.

4. Planning and Preparation for Deployment of Spack on TSCC2

Expected date of deployment: Summer 2023

• Operating system at deployment will be Rocky 9.x
• Default GCC for Rocky 9.x is gcc 11.2.x.
• All networking will be RoCE.
• There will be a Lustre filesystem.

Updates from @jerrypgreenberg.

Open questions:

• Does TSSC2 have spack user role accounts for deployment?
• Does TSSC2 ready for write access to /cm/shared?

Open Issues:

• Intermittent MPI/RoCE issues on OMB tests.

5. Upcoming work and/or development issues

• Complete initial draft of DEPLOYMENT.md, which will serve as general deployment documentation
• Complete initial draft of CONTRIBUTING.md, which will serve as documentation on how to contribute work to this repo
• Spack v0.19.2 is out; https://github.com/spack/spack/releases/tag/v0.19.2
• Create and start build out of a sdsc/spack spack-0.19.2 deployment branch for Expanse and TSSC2
• Create a dev & test spack instance(s) based on a subset of the the expanse/0.17.3/cpu and expanse/0.17.3/gpu instances
• Investigate creating a build cache on Jetstream2; may allow us to deploy more quickly during system maintenance

References:

• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8424125/Expanse+Acceptance+Testing
• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8423521/Expanse+Spack+Application+Deployment