Implement PYTHONUSERBASE in Jupyter kernel

[jsyleung]

Addresses #32 and #33

@Bai-Chiang pointed out here that PYTHONUSERBASE is preferred for being able to process *.pth files, while PYTHONPATH will not. Although they ultimately point to the same place, their declarations are not the same; PYTHONUSERBASE is intended to be the parent directory of lib/pythonX.Y/site-packages (default is ~/.local), whereas PYTHONPATH requires the full path to site-packages.

The current soconda module (v0.6.16 on Tiger3) sets PYTHONUSERBASE = $HOME/.local/soconda_v0.X.Y. Note in particular the '_' rather than the '/' prior to this commit. I believe this is from before we switched to the v0.X.Y naming scheme. For example, soconda_2025019 had $HOME/.local/soconda_20250109/3.10, which uses the '/'. It’s also possible that '/' is how it’s set up on NERSC—please check this with module show soconda.

I tested the following on Tiger3-vis with soconda v0.6.11, v0.6.15, and v0.6.16 using VS Code (1.109.3) and Jupyter extension (2025.9.1).

Prior to change

"PATH": "/usr/bin:@ENVPREFIX@/bin",
"PYTHONPATH": "${HOME}/.local/soconda/@VERSION@/lib/python@PYVER@/site-packages"

Running sys.path returns

['/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python312.zip',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12/lib-dynload',
'', '/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12/site-packages']

The user site-packages is not prepended because the '/' makes it look for soconda/v0.6.16 rather than soconda_v0.6.16.

Change '/' to '_'

Again running sys.path,

['/home/jl128/.local/soconda_v0.6.16/lib/python3.12/site-packages',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python312.zip',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12/lib-dynload',
'', '/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12/site-packages']

The prepending worked, but as expected, the packages inside *.pth files are missing (namely, sotodlib).

With PYTHONUSERBASE (this commit)

Output of sys.path

['/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python312.zip',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12/lib-dynload',
'', '/home/jl128/.local/soconda_v0.6.16/lib/python3.12/site-packages',
'/home/jl128/code/sotodlib',
'/scratch/gpfs/SIMONSOBS/env/v0.6.16/soconda_v0.6.16/lib/python3.12/site-packages']

This time sotodlib appears! (since I installed it as a *.pth) Note that, when using PYTHONUSERBASE, Python will always place the user site-packages ahead of the environments. You can verify this in Python’s site.py module: main() calls addusersitepackages() before addsitepackages().

One more question

What’s the reason for changing PATH? Loading soconda in terminal doesn’t require it, and it seems risky to replace the original PATH, which includes system libraries etc. I tested removing it and nothing seems to have changed, but perhaps I missed something?

[tskisner]

Thanks @jsyleung , I agree with the underscore typo. I did not notice that since I have been using a custom kernel file pointing to a cloned environment.

The change to PYTHONPATH was due to PYTHONUSERBASE not working at NERSC, however that was some time ago and I will test again.

For the PATH modifications, again this was because within the jupyter environment (at least at NERSC), the executables installed into our conda environment were not found (for example, from inside a jupyter terminal). However I can re-test that as well.