[NERSC] ssh doesn't work for me in desc-stack-weekly

[wmwv]

Description
A clear and concise description of what the issue is.

ssh doesn't work for me in the desc-stack-weekly environment loaded from command line.

If I use start-kernel-cli.py to load desc-stack-weekly, and then try to ssh to another node (to, e.g., monitor load, GPU usage...), I get

[login25 ~] start-kernel-cli.py desc-stack-weekly
Starting an environment for desc-stack-weekly now...
This may take a while. Be patient...
ssh lUnable to find an acceptable version of lsst_sims
[login25 ~] ssh login28
ssh: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5.so.3)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libk5crypto.so.3)
ssh: /lib64/libselinux.so.1: no version information available (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libcrypto.so.1.1)

Choose all applicable topics by placing an 'X' between the [ ]:

• jupyter
• jupyter terminal
• Cori interactive command line
• Batch jobs
• python
• CSCRATCH
• Community File System
• HPSS (tape)
• Data transfer and Globus
• New User Account or account access problems

To Reproduce

[login25 ~] start-kernel-cli.py desc-stack-weekly
Starting an environment for desc-stack-weekly now...
This may take a while. Be patient...
ssh lUnable to find an acceptable version of lsst_sims
[login25 ~] ssh login28
ssh: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libgssapi_krb5.so.2)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5.so.3)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libk5crypto.so.3)
ssh: /lib64/libselinux.so.1: no version information available (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libkrb5support.so.0)
ssh: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/udiImage/modules/mpich/dep/libcrypto.so.1.1)

[heather999]

Hi @wmwv
desc-stack-weekly is a shifter container without ssh. The primary purpose of the containers has been to provide a runtime environment rather than a full featured virtual machine. I'm not sure we need ssh from within the containers themselves (but could be coaxed into considering it). Can you just ssh to a NERSC login node from another terminal? I usually use NX when I connect to NERSC, where it's not a big deal to just open another terminal.
Take care,
Heather

[wmwv]

Sure. Won't fix is a fine answer in practice.

I think it was confusing that it triggered a GLIBC version mismatch. I thought loading the image with shifter would completely override existing paths.

[wmwv]

FWIW, the image does have ssh.

[login22 ~] start-kernel-cli.py desc-stack-weekly
[login22 ~] whereis ssh
ssh: /usr/bin/ssh /etc/ssh /opt/udiImage/bin/ssh

So /usr/bin/ssh remains in the path, but the GLIBC is no longer for correct for it. Here's the tail of the path (after all of the /opt/lsst) entries:

/opt/udiImage/modules/gpu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/udiImage/bin

If you explicitly run /opt/udiImage/bin/ssh, that works fine.