Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Nim to nimble.lock makes nimble download and compile the compiler #67

Open
rockcavera opened this issue Dec 13, 2023 · 10 comments
Open

Comments

@rockcavera
Copy link
Contributor

rockcavera commented Dec 13, 2023

I was installing nimlangserver with nimble and noticed that it always downloaded the Nim compiler, compiled csource, koch, compiler, tools... an unnecessary and slow process for those who have the compiler in the path.

"nim": {

@rockcavera
Copy link
Contributor Author

Today, after recompiling from HEAD, I checked the files that were used to compile nimlangserver and it didn't even use anything from stdlib or the Nim compiler that was installed by nimble, that is, keeping Nim in .lock only delays the process of nimlangserver compilation, as it used the Nim compiler that I already had (devel) and my devel's stdlib and not the one installed by Nimble.

@arnetheduck
Copy link
Contributor

This sounds more like a nim/nimble bug than one in langserver? ie the purpose of a lock file is to deterministically choose which libraries are used to compile the application - that quite obviously includes the standard library which makes up most of the code (and compiler, by extension).

@rockcavera
Copy link
Contributor Author

This sounds more like a nim/nimble bug than one in langserver? ie the purpose of a lock file is to deterministically choose which libraries are used to compile the application - that quite obviously includes the standard library which makes up most of the code (and compiler, by extension).

Of course, there can be several interpretations of what was intended by adding Nim to .lock. I can see two:

  1. Should the compilation be done by the Nim executable compiled by the Nimble installation when installing nimlangserver? In this case, stdlib and compiler of the Nim version informed in the .lock will be used. If this is the case, then we are facing a bug, in which Nimble should call "Nimble/pkgs2/nim-2.1.1-ee369ca2ef4fd48e55732c09c5f8b67a4a22daa4/bin/nim".

  2. However, if the requirement is to use the Nim executable previously installed on the machine, nimlangserver should import pkg/compiler/pathutils or compiler/pathutils to use the compiler package installed by Nimble. Regarding stdlib, I have no idea how to import it other than from the Nim executable that will perform the compilation.

In my opinion, the second option seems more obvious, mainly because I don't know of any package manager that installs a specific interpreter or compiler for each package you intend to install. Generally, if the interpreter or compiler does not match the version required by the installed package, the package installation should fail.

@arnetheduck
Copy link
Contributor

arnetheduck commented Jan 9, 2024

If this is the case, then we are facing a bug, in which Nimble should call "Nimble/pkgs2/nim-2.1.1-ee369ca2ef4fd48e55732c09c5f8b67a4a22daa4/bin/nim".

this is the case.

However, if the requirement is to use the Nim executable previously installed on the machine

This is also the case ;)

The key insight here is that in order to compile the application, you need to have matching versions of all dependencies - it's really that simple. If you have a system nim installation and it matches the required version, that should be used - if not, nimble should install a matching version just like it installs libraries. This same logic applies to other "tools" you might need to build the package (ie c2nim or some other custom binary of which nim is the most common example).

This enables a number of things:

  • you chances of compiling the applications shoot up significantly
  • you don't need to have a system nim at all - just a nimble (or atlas, which has similar functionality)
  • for the vast majority of packages, you will only ever install one or two nims and the correct one will automatically be used when you work on a specific package.
  • tooling like nimsuggest can also automatically pick the right compiler and std lib when you open a project in your editor of choice

Note that the above is not limited to lock files - the same logic applies if you have not locked your Nim but instead specified a version range with which it works.

don't know of any package manager

Modern package managers (ie cargo, npm etc) track the compiler/runtime version required to compile the application just like any other dependency. They also typically offer tooling to switch between versions (ie nvm, rustup, choosenim etc). The only difference you're observing is that nimble includes this functionality in the same program.

@arnetheduck
Copy link
Contributor

Oh, and finally, in nimble there should obviously be room for local overrides based on flags (--override:nim=myversion or something like this) - this applies to nim but also to any other dependency - so as an option, you should be able to override which version of a dependency is going to be used, if you are in a position where you feel like dealing with random compile errors and runtime issues because the dependency changed.

@rockcavera
Copy link
Contributor Author

Was removed in 8449dfb

@jmgomez
Copy link
Collaborator

jmgomez commented Feb 6, 2024

For the record, I removed it by mistake. But I do see the pain on downloading it every time, so my vote is to keep it as is for the time being.

@arnetheduck
Copy link
Contributor

But I do see the pain on downloading it every time

every time? Only once, it should be cached, no?

@jmgomez
Copy link
Collaborator

jmgomez commented Feb 7, 2024

Added it back. It seems I need to learn more about Nimble. I just wanted to update some deps, turned out it somewhat dumped my global Nimble folder when doing the lock (and removed Nim from there as I didnt have it)

@jmgomez jmgomez reopened this Feb 7, 2024
@melMass
Copy link

melMass commented Oct 23, 2024

Not sure if this is the cause but I couldn't get nimble to install nimlangserver on Mac using a brew installed nim (2.2.1)

nimble install nimlangserver
Downloading https://github.com/nim-lang/langserver using git
Downloading https://github.com/nim-lang/Nim.git using git
       Tip: 7 messages have been suppressed, use --verbose to show them.
nimble.nim(703)          downloadDependency

    Error:  Downloaded package checksum does not correspond to that in the lock file:
        ...   Package:           [email protected]@r.5935c3bfa9fec6505394867b23510eb5cbab3dbf
        ...   Checksum:          efb469b3ca8c4ec85c5829fa18997508672f9740
        ...   Expected checksum: 46333e8f4bda41dd6d3852a3f5fa4975b96b66a2

Cloning this repo, removing this line AND the lock file made it work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants