Add a guide/walkthrough #355

jesshart · 2023-02-18T01:17:06Z

Description

This is the draft walkthrough I mentioned on issue #350.

netlify · 2023-02-18T01:17:13Z

✅ Deploy Preview for conda-lock ready!

Name	Link
🔨 Latest commit	`f4476c5`
🔍 Latest deploy log	https://app.netlify.com/sites/conda-lock/deploys/641ca27597ccc300077de3e4
😎 Deploy Preview	https://deploy-preview-355--conda-lock.netlify.app/walkthrough
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

maresb · 2023-02-18T09:34:50Z

Thanks a lot for this!!! Quickly skimming I really like the concept. I could see this turning into some sort of quickstart guide. I think people will have a substantial amount of feedback and suggestions, but working through those may be very valuable for bridging the gap between new and experienced users.

I am also not sure if/how exactly we will publish this. That is dependent on how this evolves, and also a broader consensus among the maintainers, so I can't make any promises. Also, our review capacity is a bit strained at the moment, so I really hope that spectators step up and offer their feedback.

Thanks a lot for all the careful work and thought you have put into this!

maresb · 2023-02-18T09:38:25Z

You should probably get pre-commit running locally on your side, it is simpler than pytest.

maresb · 2023-02-18T09:38:37Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

maresb · 2023-02-18T09:41:07Z

Make sure to pull that pre-commit formatting fix before making additional commits in order to prevent merge conflicts.

timsnyder-siv · 2023-02-18T21:35:16Z

docs/walkthrough.md

+## Adding a New Package via `conda`
+We have `pandas` installed, but of course we want to use `matplotlib`.
+1. Install `matplotlib`
+```bash
+(base) $ conda install -c conda-forge matplotlib
+...
+```
+2. Add `matplotlib` to the `environment.yml` file.
+```yml
+# environment.yml
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python
+  - pandas
+  - matplotlib # NEW
+platforms:
+  - linux-64
+  - osx-64
+  - win-64
+  - osx-arm64
+```
+3. Update the file using `conda-lock`
+```bash
+(base) $ conda-lock --update matplotlib
+Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']...
+...
+```
+Note: If you update to a specific version of a package, you will need to provide the version in the command. For example, `matplotlib=3.5` should be included after the `--update` flag. This update flag is useful for the following:
+* Updating to the latest version of a package (no version specified)
+* Updating a package to a specific version
+* Adding a new package


Does this actually work? I am trying to do this same type of thing. In my case, I want to add fsspec to https://github.com/firesim/firesim/blob/main/conda-reqs.conda-lock.yml. To do that, I:

add fsspec to https://github.com/firesim/firesim/blob/main/conda-reqs.yaml

conda-lock -f conda-reqs.yaml -p linux-64 --lockfile conda-reqs.conda-lock.yml --update fsspec

NOTE: I don't necessarily install the package into the conda environment because conda-lock doesn't use the contents of the currently active environment in it's calculation of the lockfile, does it?

However, I find that unless package you're trying to add is in the lockfile, the intersection() at

conda-lock/conda_lock/conda_solver.py

Line 151 in 854fca9

to_update = set(update).intersection(conda_locked)

filters the added package out of to_update and conda-lock effectively re-solves the entire environment, it doesn't use the update code.

Also, I think it is mildly confusing to suggest step 1 is installing the package into the currently active environment because it implies that having the packages installed somehow influences how conda-lock creates the lockfile, which I don't think is correct. If you are using a lockfile to reproducibly create an environment, it would probably be better to create the updated lockfile and then use it to create the environment. I haven't tried installing on top of an existing environment to see what happens but I'd expect it to be the same as removing the environment completely and reinstalling everything in the lockfile (but perhaps with optimizations for leaving unchanged packages alone).

The INFO logger line after Locking dependencies for ... is indicative of whether update is happening or not.

When update is being used, it should say something like:

INFO:conda_lock.conda_solver:to_update: ['matplotlib']

I suspect the output looks like :

INFO:conda_lock.conda_solver:linux-64 using specs ['python', 'pandas', 'matplotlib']

and that indicates that the entire environment is being resolved and --update is effectively being ignored.

@mariusvniekerk, reading #158 it seems like incremental updating should be possible. If you could take a look at this use-case and comment on what I've found, it would be really helpful.

I can confirm that what is shown in the walkthrough is in fact ..using specs... and appears to be ignoring the --update flag. I can be sure to remove this section, or at least clarify that the environment is being solved from scratch.

I'm not very familiar with the --update flag, but if there's a suspected bug (or a missing warning message), let's open an issue for follow-up.

Thanks @mariusvniekerk. I guess I was asking whether it is supposed to be possible to add new specs to the solve and ask to only update those new specs or if one currently must re-solve the whole environment.

That should work i think, but yeah, probably worth testing if it behaves in the way that you'd expect

The more standard approach is to just delete and get a new solve

I can modify the document to recommend this approach. However, to your point,

That should work i think, but yeah, probably worth testing if it behaves in the way that you'd expect

It appears the --update flag is not behaving as expected. For example, when I used it to add matplotlib it behaved like this:

(base) $ conda-lock --update matplotlib INFO:conda_lock.conda_solver:linux-64 using specs ['python', 'pandas', 'matplotlib']

This is in contrast to what was expected according to @timsnyder-siv:

INFO:conda_lock.conda_solver:to_update: ['matplotlib']

So it appears the command is ignoring the flag.

@jesshart we should probably open a new issue referencing this. If we're feeling strong in our python-fu, PR'ing a test would also be helpful and if we're feeling particularly masterful, possibly a patch that helps conda-lock do this solve. I looked at the code enough to think I might be able to come up with a patch but it will probably take me longer to get right than I have time for right now. I'll open a ticket from this thread.

Opened #386 for this.

mariusvniekerk · 2023-02-24T20:06:54Z

docs/walkthrough.md

+Sometimes a package installed via `conda` (even using the `conda-forge` channel) breaks my environment. I have found using `pip` as the *exception to the rule* has resolved most of these issues.
+
+### Why is this process slow?
+Solving the dependencies of dependencies for one operating system is time consuming. Out of the box,  `conda-lock`  will try to solve for 3 different operating systems (i.e., linux-64, osx-64, and win-64). However, `conda` recently announced partnering with a `mamba` solver. You could configure your `conda` instance to use a faster solver like this one. This is beyond the scope of this walkthrough. Regardless, it will still take time.


conda-lock defaults to using mamba if its installed so we should just recommend that.

I have been using miniconda with the libmamba solver enabled (ref). Is that much different from using mamba directly?

they're similar but not the same

Not trying to be obtuse here, but wouldn't a strong dependency like this discourage a variety of users from engaging conda-lock? I am happy to adapt this documentation, but wanted to approach this philosophical point first.

For example, I could make the point in the documentation so readers are not misled into thinking it only works with the mamba setup.

jesshart · 2023-03-13T21:04:35Z

Document Changes

Replace libmamba approach with a mamba approach (different from utilizing libmamba in conda)
Remove references to --update flag since it is not working as expected (--update ignored for packages not in existing lockfile #386)

…h-doc

jesshart · 2023-03-15T16:16:48Z

@maresb I updated the document per the conversations here. However, I forgot to follow your instructions,

Make sure to pull that pre-commit formatting fix before making additional commits in order to prevent merge conflicts.

Which may have caused a problem. I resolved the merge conflicts on my end, but it looks like they still broke during the pre-commit.ci - pr step. Do I need to do anything on my end?

maresb · 2023-03-15T17:33:35Z

@jesshart, if you like your local version, then the easiest solution is to do a force-push git push --force and this will replace the conflicting walkthrough-doc branch here with your local copy.

mariusvniekerk · 2023-03-23T19:02:25Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

create file with draft pasted

dab04ff

jesshart requested a review from a team as a code owner February 18, 2023 01:17

jesshart changed the title ~~create file with draft pasted~~ resolves #350 Feb 18, 2023

maresb changed the title ~~resolves #350~~ Add a guide/walkthrough Feb 18, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

6c3f2d8

for more information, see https://pre-commit.ci

timsnyder-siv reviewed Feb 18, 2023

View reviewed changes

timsnyder-siv mentioned this pull request Feb 21, 2023

Add URI support to tarball path and xclbin path firesim/firesim#1432

Merged

12 tasks

mariusvniekerk reviewed Feb 24, 2023

View reviewed changes

timsnyder-siv mentioned this pull request Mar 6, 2023

--update ignored for packages not in existing lockfile #386

Open

jhartman and others added 5 commits March 14, 2023 15:21

replace conda with mamba

d1359ae

Merge remote-tracking branch 'origin/walkthrough-doc' into walkthroug…

a496b44

…h-doc

remove defaults channel from conda approach artifact

36406d7

fix update section and make other minor edits

bb2ced3

Merge branch 'conda:main' into walkthrough-doc

9eb95d5

remove table of contents, improve number reference

557bd7a

[pre-commit.ci] auto fixes from pre-commit.com hooks

f4476c5

for more information, see https://pre-commit.ci

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a guide/walkthrough #355

Add a guide/walkthrough #355

jesshart commented Feb 18, 2023

netlify bot commented Feb 18, 2023 •

edited

Loading

maresb commented Feb 18, 2023

maresb commented Feb 18, 2023

maresb commented Feb 18, 2023

maresb commented Feb 18, 2023

timsnyder-siv Feb 18, 2023

timsnyder-siv Feb 18, 2023

timsnyder-siv Feb 21, 2023

jesshart Feb 22, 2023

maresb Feb 22, 2023

timsnyder-siv Feb 22, 2023

mariusvniekerk Feb 24, 2023

jesshart Feb 24, 2023

timsnyder-siv Mar 6, 2023

timsnyder-siv Mar 6, 2023

mariusvniekerk Feb 24, 2023

jesshart Feb 24, 2023

mariusvniekerk Mar 7, 2023

jesshart Mar 13, 2023

jesshart commented Mar 13, 2023 •

edited

Loading

jesshart commented Mar 15, 2023

maresb commented Mar 15, 2023

mariusvniekerk commented Mar 23, 2023

Add a guide/walkthrough #355

Are you sure you want to change the base?

Add a guide/walkthrough #355

Conversation

jesshart commented Feb 18, 2023

Description

netlify bot commented Feb 18, 2023 • edited Loading

✅ Deploy Preview for conda-lock ready!

maresb commented Feb 18, 2023

maresb commented Feb 18, 2023

maresb commented Feb 18, 2023

maresb commented Feb 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jesshart commented Mar 13, 2023 • edited Loading

jesshart commented Mar 15, 2023

maresb commented Mar 15, 2023

mariusvniekerk commented Mar 23, 2023

netlify bot commented Feb 18, 2023 •

edited

Loading

jesshart commented Mar 13, 2023 •

edited

Loading