[NERSC] Add DESC product

[Catarina-Alves]

Description
I would like to add the DESC product snmachine to NERSC so anyone could use it in their analysis. What should I do it include it there?

Choose all applicable topics by placing an 'X' between the [ ]:

• jupyter
• jupyter terminal
• Cori interactive command line
• Batch jobs
• python
• CSCRATCH
• Community File System
• HPSS (tape)
• Data transfer and Globus
• New User Account or account access problems

[heather999]

Hi @Catarina-Alves Thank you for reaching out about snmachine. This looks like a request to add snmachine to desc-python, which is a collection of useful DESC packages, installed at NERSC (and IN2P3). desc-python is available to use both in jupyter and via command-line or batch jobs. I'm out on vacation today - but will look at this on Monday. We are trying to transition to a more formal process to include packages in desc-python, with a very light set of requirements such as tagged releases, clear installation instructions, and some demonstration that our automated build of desc-python is compatible with both new and updated packages.
It looks like snmachine is tagged 2.0.0 and is available for installation via conda. I need to set up a build of desc-python which includes snmachine and we can discuss this further next week!

[Catarina-Alves]

Thank you, I look forward to it.
snmachine is also instalable through pip, if that is easier to install.

[Catarina-Alves]

Hi @heather999, I just wanted to follow up on this.
Do you have a timeline as to when this might be ready? I do not need this before September but it would be great to have snmachine in NERS in September.

[heather999]

Hi @Catarina-Alves I'm really hoping to have snmachine in desc-python this week and certainly before September. Once I have it included - can you recommend an easy way to test it? I suspect it'll be fine, but having some way to confirm that it's working properly would be helpful. I'll let you know as soon as it is available!

[Catarina-Alves]

Thank you.
snmachine has some example notebooks (see here). If these are running without problems, snmachine should be working properly.

[heather999]

HI @Catarina-Alves I made an initial test of installing snmachine and took a look at the instructions here: https://lsstdesc.org/snmachine/install.html#using-conda
It looks like the snmachine dependencies need to be installed manually, and that doing pip install snmachine doesn't pull in all the packages needed, like lightgbm? Typically, the pip install will also install required dependencies and could even have optional dependencies that are installed for a "full" installation versus a minimal one.
I'm guessing all the dependencies are in: https://github.com/LSSTDESC/snmachine/blob/main/environment.yml.
To include snmachine in desc-python, it needs to be able to fully install itself and its required dependencies, otherwise we'd be left trying to manage snmachine's dependencies in desc-python itself, rather than have snmachine dictate what it needs.
I think this is a relatively easy fix though, the setup.py or setup.cfg could include an install_requires directive that lists what snmachine depends on. sncosmo has an example: https://github.com/sncosmo/sncosmo/blob/master/setup.cfg

[Catarina-Alves]

Hi @heather999, thank you, I tried to fix the issue.
Is the new setup in here good, or do I need to do something different?

[heather999]

Hi @Catarina-Alves Thank you - that's a start, I'm not sure this is a list of what snmachine minimally needs to run correctly. I'm wondering if something like jupyter is really required or just useful to have if you're running a notebook. Ultimately, we will also need this updated snmachine to be tagged and uploaded to pypi -that way when I go to do pip install snmachine, all the dependencies will be recognized and installed as well.

[Catarina-Alves]

Thank you @heather999.
I have a question regarding the first part of your answer. Should I only include in install_requires the base packages the snmachine uses? If so, do I leave the other packages in environment.yml like they currently are?
And how basic do I need to go? jupyter is only useful if you want to run a notebook, but we have notebook tutorials in snmachine. Similarly with the plotting packages (e.g. matplotlib, seaborn) are only used for making plots.
On a different note, I can remove the python>=3.7 install_requires, and add it to python_requires=">=3.7".

Regarding the second part of your answer. I will tag and upload to pypi after solving this issue.

[heather999]

Hi @Catarina-Alves
the setup.py and environment.yml are serving two different purposes. The setup.py is really just for installing your package and the minimal dependencies snmachine needs to run. The environment.yml is a file that lists all the packages your group is recommending a user might want to install - so it includes "extra" things like jupyter so people can run your notebooks.
Up to now, it looks like in the snmachine documentation, users need to install all the dependencies plus useful packages on their own using environment.yml and then install snmachine.
Instead of doing that, the more typical thing to do would be to update setup.py and update the snmachine on PyPI to allow users to install snmachine completely, including the required dependencies - just enough that they can use snmachine. The environment.yml could be updated to add additional useful packages - but has no direct implications for the snmachine install for a user doing: pip install snmachine. Doing that pip install should be enough for someone to use your package in most cases.

I'm going to make some suggestions on your open snmachine github issue to help define a set of packages that you absolutely need.

[heather999]

Hi @Catarina-Alves
I see the update was merged to snmachine - so next would be to update PyPI so we can install this version in the DESC python environments. I'd like to open a PR on snmachine to suggest this could be an automated procedure, so that future releases of snmachine are pushed to PyPi and made available. I"ll do that now and perhaps we can work through the steps to store the PyPI token needed to allow this to work?

[Catarina-Alves]

Hi @heather999 I just wanted to know if you found any issues, or if snmachine was able to be integrated into desc-python (thus closing this issue and issue LSSTDESC/snmachine#255).

[heather999]

hi @Catarina-Alves Sorry for the delay! I haven't found any issues. I'm working to get a new version of desc-python out, first as desc-python-dev which will include snmachine and then by the end of the week, I'm hoping to release a new desc-python with snmachine. I'll keep you updated on progress!!

[Catarina-Alves]

Thank you. I am glad that there were no issues

[heather999]

@Catarina-Alves sorry for the long delay! There is now a new desc-python-dev which includes snmachine and updated versions of a variety of packages like CCL, sacc, etc. Please feel free to try this out, if you have time.
On Thursday, I will be updating desc-python with this new version so that the desc-python environment will itself have snmachine available. Details for how to use the new desc-python-dev are on #desc-nersc: https://lsstc.slack.com/archives/C2U2K05JR/p1634598752031000
and summarized here:
To use the new desc-python-dev in Jupyter, you must update your DESC jupyter kernels by running (one time only) at the Cori command line:
source /global/common/software/lsst/common/miniconda/kernels/setup.sh
To run at the Cori command line you would do:
source /global/common/software/lsst/common/miniconda/setup_dev_python.sh
and then you will be in the desc-python-dev conda environment

[Catarina-Alves]

Hi @heather999, thank you.
I can see snmachine in desc-python-dev and use its functions. However, I can only see the files that are below this level: https://github.com/LSSTDESC/snmachine/tree/main/snmachine . Are the other files not used/needed? For example, I do not see utils folder and its scripts nor the example notebooks.

[heather999]

Hi @Catarina-Alves
snmachine and utils are both there in desc-python-dev and now in desc-python too, but since they are defined as two packages to pypi pip, they are installed in separate directories. Maybe util (esp given its very generic name) should be renamed or made a subpackage of snmachine and moved under the snmachine directory in your github repo.

Here is snmachine in desc-python:

heatherk@cori04:/global/common/software/lsst/common/miniconda/prod/envs/desc/lib
/python3.8/site-packages/snmachine> ls
__init__.py  chisq.py      parametric_models.py  sndata.py
__pycache__  example_data  snaugment.py          snfeatures.py
analysis.py  gps.py        snclassifier.py       tsne_plot.py

and here is utils:

heatherk@cori04:/global/common/software/lsst/common/miniconda/prod/envs/desc/lib
/python3.8/site-packages/utils> ls
__init__.py  imblearn_augment.py              plasticc_pipeline.py
__pycache__  plasticc_feature_engineering.py  plasticc_utils.py
config.yml   plasticc_make_predictions.py

so both packages are "top-level". I do think that utils could be made a subpackage of snmachine, which might make more sense. For now I think this works, but it could be something to think about for an upcoming release.

You're right the example notebooks are not included in the PyPI distribution. I'm curious what others think but I would rather users copy the notebooks from github. Pinging more knowledgeable folks than I for their opinions @beckermr & @rmjarvis.
I see CCL now has a separate CCLX repository for its notebook examples. A separate repo might be helpful for both your notebooks and the example data.

[beckermr]

No comment here on the notebooks but naming your python project utils is asking for trouble.

[Catarina-Alves]

I thought everything would be included in snmachine. utils was supposed to simply be a folder in snmachine that had some utility scripts. When I look at "Your projects" in PyPI, I only see snmachine

[Catarina-Alves]

I am confused with what is desirable to include when packaging a library. At the moment the snmachine repository has several folders, including snmachine, utils, tests, and examples.
I see that in the setup.py I include snmachine and utils as separate packages. As @heather999 suggested, I do think it makes more sense to make utils a subpackage of snmachine (a subfolder of that folder). But should I also include the tests? Or are these only relevant for developers and thus can remain in the GitHub without being in the setup.py?
Similar question regarding the notebook examples

[heather999]

All good questions!

• I'll open an issue on snmachine and we can make utils a subpackage
• I think tests should be included, because they provide a way for people unfamiliar with your package an easy way to make sure things are working as expected. When I say test though, I mean things that can be run at the command line, not jupyter notebooks.
• I am not inclined to distribute notebook examples as part of a pypi project for a few reasons
  • notebooks tend to change and get updated more quickly than the actual software
  • a user will likely want to edit and modify those notebooks to play around and I wouldn't necessarily want to do that with files in my conda installation.
  • But I am interested to hear about other opinions on this - it may be more a philosophical question rather than a rule.

[johannct]

furthermore conda/pip installation typically install stuff away from the user work directory (under a virtual env directory in .local or under the conda distrib etc...) so quickly it becomes impractical to leave them there. Likewise for tests actually, though I understand the point of having them around..... one can actually pytest on them, but one has to find them and likely chdir first (running tests can also suppose a certain way to preload the packages/module that may be different than the install final system if I am not mistaken).

[Catarina-Alves]

snmachine has now utils as a subpackage. This change was also propagated to pip install (v2.0.0.post3). Is there anything else to do, or can we close this issue @heather999?

[heather999]

Hi @Catarina-Alves Great!
I think the final point is that after conferring with the Johann and Gautham (now a few months ago!) - we'd like snmachine to be part of the sn-py environment rather than desc-python. It seems to be a better fit.
I have a new sn-py environment set up at NERSC which also includes the LSST Sci Pipelines and all the packages that I'm aware of that are of interest to the SN group. Some ongoing discussion on this issue.
Does that sound ok to you?

[Catarina-Alves]

Hi @heather999, yes, it seems great. Thank you

[heather999]

HI @Catarina-Alves
Finally back to this - there is a new td_env now available at NERSC which includes snmachine.
For now this environment is still a test, so to use it at the Cori command line you would do:
source /global/cfs/cdirs/lsst/groups/TD/setup_td_dev.sh
Then you're in the environment and will have access to the LSST Sci Pipelines, plus snmachine and a few other packages of interest to SN and SL.

There is also a new jupyter kernel for this environment available named desc-td-env.
To enable it, you would redo the DESC kernel setup you likely have already done in the past:
source /global/common/software/lsst/common/miniconda/kernels/setup.sh
That will re-install all the available kernels (like desc-python, desc-stack, etc) and add this new desc-td-env
https://confluence.slac.stanford.edu/display/LSSTDESC/Using+Jupyter+at+NERSC#UsingJupyteratNERSC-setup

The next time you start up a fresh jupyter.nersc.gov, the desc-td-env should be one of the available kernels.

We're hoping to make this new td_env the "stable" TD environment in the next two weeks - so please give it a try and let me know if you have trouble or success!

[Catarina-Alves]

Hi @heather999,

I find a problem when I import the modules sndata and snfeatures from snmachine; they require sncosmo, which is not part of the desc-td-env. sncosmo is part of the requirements of snmachine; you can see it in environment.yml. I had no problems with the other snmachine modules so far.

[heather999]

Thank you @Catarina-Alves
I fixed the problem - and will get a new install of the environment set up for you to test! I'm stuck with a hopefully transient issue re-installing the underlying LSST Sci Pipelines environment, but hopefully that will be sorted out by the morning.

[heather999]

Hi @Catarina-Alves
There were a few DNS problems at NERSC that delayed this.. but there is now a new build of the td_env that includes 'sncosmoas well assnmachine. You can use this updated environment at NERSC by doing: source /global/cfs/cdirs/lsst/groups/TD/setup_td_dev.sh`
Please give it another try and let me know how it goes!

[Catarina-Alves]

I updated the environment but I now find a strange problem. When I run from snmachine import snfeatures, the kernel dies. I am not sure why because the other modules are loaded without any issue. Can you also reproduce this strange behaviour?

[heather999]

Hi @Catarina-Alves I do see the same error and I traced it back to this import pymultinest here: https://github.com/LSSTDESC/snmachine/blob/abe55bac3154311477f108861274b3e453bc2285/snmachine/snfeatures.py#L31
This conda environment has mpi4py installed and built against the NERSC cray mpich libraries - which really only works on the batch nodes, not the login nodes or in jupyter where we typically run on login nodes as well. This was discussed in an old issue on the pymultinest github repo: JohannesBuchner/PyMultiNest#113

pymultinest has an optional dependency on mpi4py, it doesn't absolutely need it to run, and the code is written to check if importing mpi4py works, and if it doesn't, then the mpi4py features are just not available. In this case at NERSC, the attempt to import mpi4py results in a fatal error which the exception handling in pymultinest doesn't catch. I have some ideas about how to deal with this.. hope to know more in the morning.

[Catarina-Alves]

I do not use pymultinest often on snmachine but it is an option available to users.

[heather999]

I've opened an issue on pymultinest: JohannesBuchner/PyMultiNest#209 with one potential way to work around this.
In the meantime, I'll see if I can modify the set up so that this doesn't cause a fatal error and pymultinest will just continue without mpi on the NERSC login nodes.

[heather999]

@Catarina-Alves I have a work-around in place that prevents MPI from being used if we run python from a login node or jupyter session, but allows MPI if we're on a batch/compute node. I'm now able to successfully execute: from snmachine import snfeatures. Please give it another try when you get a chance and let me know how it goes!

[Catarina-Alves]

Seems all right now, thanks

[github-actions]

This issue is stale because it has been 90 days since last activity.
If no further activities take place, this issue will be closed in 14 days.
Add label keep to keep this issue.