Spack Working Group @ SDSC - Meeting - 2023/04/14

[mkandes]

Attendance

• Marty Kandes
• Mahidahr Tatineni
• Subhashini Sivagnanam
• Jerry Greenberg
• Nicole Wolter
• Yuwu Chen
• Trevor Cooper

Agenda

1. Upcoming Planned Maintenance on Expanse

Expected date of maintenance: May 8th, 2023

• Update operating system from Rocky 8.5 to Rocky 8.7
• Update Infiniband driver from Mellanox OFED 4.9-4.1.7 to Mellanox OFED 5.8-1.1.2
• Update Lustre client from lfs 2.12.8_6_g5457c37 to lfs 2.15.2
• Update NVIDIA GPU driver from NVIDIA 510.39.01 CUDA 11.6 to NVIDIA 515.65.01 CUDA 11.7

Modifications made to the expanse/0.17.3/cpu and expanse/0.17.3/gpu instance build specs and configurations made during testing on devexp are currently being committed back to the sdsc-0.17.3 deployment branch in preparation for deployment on the updated production nodes currently available on Expanse under ReservationName=rocky8u7_testing.

Prior to deployment of the new instances as expanse/0.17.3/cpu/b and expanse/0.17.3/gpu/b, a few additional modifications will be made. These include:

• Finish removing architecture-specific designations on external packages in packages.yaml
• Explicitly set default modules using spack module lmod setdefault; open question: if this command is run as part of the spec build script, what happens when another variant is installed? Does the default behavior again override and set last installed as default? i.e., will we need to retrospectively set defaults over and over again, or is default designation "sticky"?
• Move spec build scripts for "broken" packages that won't build, but we may want to reconsider in the future for production deployment into a designated subdirectory in their build layer to make it more clear they should not currently be considered a part of the package dependency chain.
• Remove spec build scripts for packages that are no longer or currently not needed

2. Update on pre-production testing of expanse/0.17.3/cpu/a and expanse/0.17.3/gpu/a

Ready to close:

• None

Ready for testing:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing matlab (example application) #39
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gamess (example application) #36
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing cp2k (example application) #35
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing QE (example application) #63
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing NWCHEM (example application) #62
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing NEURON (example application) #61
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing NAMD (example application) #60
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gromacs(example application) #58
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing LAMMPS (example application) #55
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing gromacs GPU(example application) #54

Open runtime problems:

• ATS-2095/ZEN-30304 - expanse/0.17.3/gpu/b - Missing LAMMPS GPU (example application) #56

Open build problems:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gaussian (example application) #37
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing Amber GPU(example application) #50
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing NAMD GPU (example application) #59
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing gaussian (example application) #57

Still need to build test:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Wannier90 (example application) #46
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Abinit (example application) #33
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Abaqus 2018, 2021 (example application) #32
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing matlab (example application) #67
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing Amber GPU(example application) #50

Updates from @nwolter.

3. Planning and Preparation for Deployment of Spack on TSCC2

Expected date of deployment: Summer 2023

• Operating system at deployment will be Rocky 9.x
• Default GCC for Rocky 9.x is gcc 11.2.x.
• All networking will be RoCE.
• There will be a Lustre filesystem.

Updates from @jerrypgreenberg. Are we ready to checkpoint tscc/0.17.3/cpu and tscc/0.17.3/gpu and integrate them in to the main sdsc/spack repo? If not, when should that occur?

Open questions:

• Does TSSC2 have spack user role accounts for deployment?
• Does TSSC2 ready for write access to /cm/shared?

4. Upcoming work and/or development issues

• Complete initial draft of DEPLOYMENT.md, which will serve as general deployment documentation
• Complete initial draft of CONTRIBUTING.md, which will serve as documentation on how to contribute work to this repo
• Spack v0.19.2 is out; https://github.com/spack/spack/releases/tag/v0.19.2
• Create and start build out of a sdsc/spack spack-0.19.2 deployment branch for Expanse and TSSC2
• Create a dev & test spack instance(s) based on a subset of the the expanse/0.17.3/cpu and expanse/0.17.3/gpu instances
• Investigate creating a build cache on Jetstream2; may allow us to deploy more quickly during system maintenance

References:

• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8424125/Expanse+Acceptance+Testing
• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8423521/Expanse+Spack+Application+Deployment

[mkandes]

@mkandes - Reminder, we need to open a Jira issue to update the Spack instance modules.

[tcooper]

The basic config is already commited and deployed to the 6 test nodes. It is part of a PR that will get merged before the PM. Details are here for you to link in the issue... https://hpc-gitlab.sdsc.edu/hpc-operations/common/bright-ansible/-/merge_requests/13/diffs?commit_id=a4a12d67f06cde0c27d2f3fb3693bd829d633f3b The issue can still be created and will cover items such as content of Description and others. Eg... ~ ❯ module whatis cpu/0.17.3a cpu/0.17.3a : Name : cpu cpu/0.17.3a : Version : 0.17.3 cpu/0.17.3a : Release : a cpu/0.17.3a : Target : zen cpu/0.17.3a : Short Description : Adds modules for SDSC Software Stack for AMD / CPU Applications to MODULEPATH. ~ ❯ module whatis gpu/0.17.3b gpu/0.17.3b : Name : gpu gpu/0.17.3b : Version : 0.17.3 gpu/0.17.3b : Release : b gpu/0.17.3b : Target : skylake_avx512 gpu/0.17.3b : Short Description : Adds modules for SDSC Software Stack for Intel / GPU Applications to MODULEPATH. I think you get the picture. Trevor Cooper, M.Sc. HPC Systems Programmer San Diego Supercomputer Center GPG Fingerprint: 2CA999800D11C5946C9DBFEE52364D7BBCEB35B8 GPG Key: download here<https://users.sdsc.edu/~tcooper/TrevorCooper.pgp>

…

________________________________ From: Marty Kandes ***@***.***> Sent: Friday, April 14, 2023 1:21 PM To: sdsc/spack ***@***.***> Cc: Cooper, Trevor ***@***.***>; Mention ***@***.***> Subject: Re: [sdsc/spack] Spack Working Group @ SDSC - 2023/04/14 (Discussion #68) @tcooper<https://urldefense.com/v3/__https://github.com/tcooper__;!!Mih3wA!AIIIWrCimMRZNNYq6xwMuupTuyR-mbjyDujI_S3s8nIR0Up3weRga33VsmdNbeixeGaB__DNNOAI1V6ODD73MSPj$> @mkandes<https://urldefense.com/v3/__https://github.com/mkandes__;!!Mih3wA!AIIIWrCimMRZNNYq6xwMuupTuyR-mbjyDujI_S3s8nIR0Up3weRga33VsmdNbeixeGaB__DNNOAI1V6ODIZRR_Fr$> - Reminder, we need to open a Jira issue to update the Spack instance modules. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/sdsc/spack/discussions/68*discussioncomment-5619990__;Iw!!Mih3wA!AIIIWrCimMRZNNYq6xwMuupTuyR-mbjyDujI_S3s8nIR0Up3weRga33VsmdNbeixeGaB__DNNOAI1V6ODGWJvE7d$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AAEEQ7XBSECMFLIVTDNFFVDXBGWWBANCNFSM6AAAAAAW626CDA__;!!Mih3wA!AIIIWrCimMRZNNYq6xwMuupTuyR-mbjyDujI_S3s8nIR0Up3weRga33VsmdNbeixeGaB__DNNOAI1V6ODDb39ZFv$>. You are receiving this because you were mentioned.Message ID: ***@***.***>