Setting up new desc-pipelines conda-forge metapackage #47
Comments
|
I'm happy to work on conda-forge packages for many of these things. A few notes.
Some of the authors may need to go back and address these items. Bump (@yymao, @fjaviersanchez, and others I am missing) |
|
@beckermr are we going to have science pipelines v19 (or 20?) in conda-forge (via stackvana?). If so, I can put |
|
I have not pushed I am more than happy to review PRs for any conda-forge package by DESC members, so do feel free to make a PR when you are ready. Submit them here (https://github.com/conda-forge/staged-recipes) and bump me when you are ready. |
|
Note that some packages may not want a desc prefix. For example, we did not add one for CCL and adding one for things like |
|
Some packages may have names that are too generic. It doesn't have to be a desc prefix, but some packages may want to ponder their future name... like |
|
If it's easy to |
|
Hmm, I am not sure if I have the time/resources to make my packages listed above into conda-forge. Are they all required by this new desc-pipelines metapackage? |
|
@yymao I am happy to make conda-forge packages for whatever we'd like. I don't want to go around releasing versions though. We don't need to move all of them. even having a rough metapackage in conda-forge would help DESC users standup environments in various places that are close to what is at nersc. |
|
Thanks, @beckermr. In Heather's list, packages that I am the primary maintainer of all have versioned releases, and all have open source licenses (MIT, and in one case GPL), so I think they already satisfy Matt's (1) and (4). Below I list their dependency situation: General-purpose packages
DESC-specific packages
|
|
Thanks for the ping @beckermr! I added a license (BSD 3 Clause) and tagged CatalogMatcher. This package depends on |
|
Hi @beckermr Sorry for the long silence. Using the rubinenv-feedstock example and the general instructions in the conda-forge doc - I went ahead and forked conda-forge/staged-recipes, made a branch and added a first stab at a recipe for a lsstdesc-env, based on what we have in desc-python and leaving off what is just available via pip. I tried a local build and nothing horrible happend! I might be brave enough to open a PR and see where this goes :) https://github.com/heather999/staged-recipes/tree/lsstdesc-env/recipes/lsstdesc.env |
|
Looks great. One minor comment We should remove this line and add to the build section. This will make the recipe installable on any platform. If some of the deps are not available, then it simply won't install which is fine. As far as the package name, the rubin folks made a mistake when making the feedstock. I'd make both lsstdesc-env. We should also decide on a versioning scheme for the env now. |
|
We could also go with |
|
While I like the dash, I'm happy to stick with the Open to suggestions on the versioning scheme.. I was just going with major.minor.patch and starting with 0.Y.Z for now - since this is all preliminary. That was about as far as I thought about it :) One question, I noticed rubinenv included their own conda_build_config.yaml and found: https://conda-forge.org/docs/maintainer/knowledge_base.html#using-centos-7 but also https://conda-forge.org/docs/maintainer/infrastructure.html#centos-sysroot-for-linux-platforms |
Right. So usually people use semver to set API expectations. Is that the case here? Do we want to use a version that is a straight date?
You do not need this. This was done to support centos7 system libs for parts of the DM stack. We won't need that or we will let the stack pull that in itself. |
|
Another thing. We can trade the dot for a dash in package names now if you'd like. I got the dot from @rmjarvis and lsstdesc.coord. I wonder if he has options here. |
|
For versioning - lsstdesc.env itself has no API but some of the included packages will - does that need to be expressed? Dates will make it abundantly clear how old/new a version is, but I wonder if it will be easy to refer to down the road.. would people easily recall something like 2020.01.14 (or 2020.01.14_micro if there's some chance there is a quick patch on the same day)? I'm always worried about confusing colleagues with dates if it's unclear what is referring to the month and what is the day.. but I think YYYY.MM.DD is pretty common. I do prefer the dash, but willing to yield to other opinions on that. I figure the most important thing is consistency. |
|
*opinions |
Right, so if we were to increase a major or minor version each time an underlying package was updated, we'd be pushing them all the time. I prefer a date system myself. Also remember this is an abstract metapackage. It specifies what to install but not necessarily which versions. That is left to conda and the underlying constraints of the packages themselves. So we shouldn't worry too much. |
|
What are you asking for opinions about? |
|
We were asking about |
|
I personally prefer the |
|
KK @heather999 let's go with a hyphen. I will change the other packages we have to use both in order to keep things backwards compatible. |
|
Ok - updated to use the hyphen! As for version, sure let's go with dates: YYYY.MM.DD Any need to initially indicate a |
|
For this, I am not too worried about patch numbers. The use of this env since it contains abstract specs is to help people easily bootstrap an env with the right packages locally. Others should weigh in here. I know what I would like but other people might have better ideas. |
|
Ok - let's move forward, how about |
|
Yup. See the package conda-forge/conda-forge-pinning-feedstock. We can autogenerate the version. |
|
We could also append the build number to the version. |
|
ah yes, autogenerating the version - I was wondering about that! Ok - let's see if I can get this in. I like the idea of appending the build number too - I'll just out and ask how to do that, is that stored in |
|
make a jinja2 var and then add it to the version. I can make the change on staged recipes if you make a PR there. |
|
Great, thanks! I submitted the PR and see the failure for win64. Just digging around, starting with |
|
noarch recpes only build on linux. You can ignore the windows failure. |
|
Well that's done! So now we sit back and suddenly |
|
We don't have to rename the repos. We can make the conda name different than the repo name. |
|
I've made easyquery into conda-forge (https://github.com/conda-forge/easyquery-feedstock) 🎉. Since easyquery was already in PyPI, grayskull makes it easy to create the recipe. The conda-forge community is also very helpful. Now that easyquery is in, I think GCR can be done relatively straightforward as well, since it's also in PyPI and we can use grayskull. For fast3tree, while it is in PyPI, the C library part may makes it not as straightforward. Other packages in my list above are not in PyPI and I have not learned how to generate recipes for them yet. |
|
I can tackle the ones with C builds. We'll need the proper license files from rockstar for fast3tree |
|
And yes the conda-forge community is a really nice one. |
|
@yymao I opened an issue on fast3tree to help guide changes to the code base to make the installation more robust and also easier for conda: yymao/fast3tree#3 We will likely need to address this issue first before we can properly build it in conda-forge. |
|
For recipes not in pypi, you can pull the source from the github release but you will need to write the meta.yaml by hand. I can start in on those. |
|
Thanks @beckermr! When I was tidying up the fast3tree repo yesterday, I had a feeling what you said in yymao/fast3tree#3 would be an issue. There's however a complicating factor about how the so files are generated... I'll ask you some questions on Slack about this later. |
|
@beckermr A question just came up from @cwwalter about the ability to search for all DESC packages available in conda-forge. We discussed using that |
|
We decided that for DESC specific packages, we would use the prefix, but that for more general things, we would not. You can use |
|
How do you list all the packages that are in |
|
You look at the recipe. |
|
BTW, GCR is now on conda-forge. Today I'll try to move GCRCatalogs there too. Based on the discussion here, I'll call it |
|
The name should be |
|
I agree that would be easiest to keep track of. Guess there is the question of what is general versus what is specific to DESC. I think code that lives in the LSSTDESC GH organization likely should always utilize the |
|
I don't know if that's always the case, but I'm not sure. Right now |
|
I agree. We leave this distinction up to the authors, like everything else. |
|
So going back to Chris' question - what would we suggest to find a list of all "DESC" packages in conda-forge? I just happen to be aware of what's available from experience, but that's not terribly useful for someone else. Asking users to look at a recipe may not be the most friendly option. After install of something like lsstdesc-env, one can list the packages - I'm not aware of a way to get conda to list package dependencies before install (other than during the install). |
|
Here you go |
|
The version weirdness is a bug: mamba-org/mamba#618 |
|
In terms of finding all DESC packages, there is not a great way to do this unless we force people into names which we should not and goes against our culture IMHO. We don't want to install everything at once anyway. If you need something, you'll know the name and then conda will install the dependencies. |
|
We could use this repo with a software-catalog.md or the like and just encourage people to add to that file when they add things to conda-forge. Of course, you can always search this github org (for those packages hosted on it) and we can also recommend people add reference to packages (and their names) when they are added to conda-forge in the README.md |
|
This issue is stale because it has been 90 days since last activity. |
As discussed on Slack: https://lsstc.slack.com/archives/G8BV3R04C/p1603276418046700
Plan is to create a new desc-pipelines metapackage in conda-forge. Need to collect a full list of expected packages and looking for input!
Expectation is that the package list will be based on what is already in desc-python
Note some of these packages may only available via pip:
May want to add some additional packages:
Other suggestions?
The text was updated successfully, but these errors were encountered: