Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature request: support the compressed (non-extracted) version of the Pkg server registry #442

Closed
CameronBieganek opened this issue Feb 28, 2022 · 8 comments · Fixed by #568
Closed

Feature request: support the compressed (non-extracted) version of the Pkg server registry #442

CameronBieganek opened this issue Feb 28, 2022 · 8 comments · Fixed by #568

Comments

@CameronBieganek
Copy link 

julia> all_pkg_names = AutoMerge.get_all_non_jll_package_names("/Users/cameron/.julia/registries")
ERROR: "/Users/cameron/.julia/registries/Registry.toml": No such file
Stacktrace:
 [1] error(::String, ::String)
   @ Base ./error.jl:42
 [2] readstring
   @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/TOML/src/TOML.jl:21 [inlined]
 [3] parsefile(f::String)
   @ TOML /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/TOML/src/TOML.jl:43
 [4] get_all_non_jll_package_names(registry_dir::String)
   @ RegistryCI.AutoMerge ~/.julia/packages/RegistryCI/Nj1Mv/src/AutoMerge/util.jl:285
 [5] top-level scope
   @ REPL[7]:1

Here are the contents of my ~/.julia/registries directory:

total 4.2M
-rw-r--r-- 1 cameron staff 4.2M Feb 27 20:51 General.tar.gz
-rw-r--r-- 1 cameron staff  129 Feb 27 20:51 General.toml
@DilumAluthge
Copy link
Member

There are three possible formats that the registry can take. Unfortunately, RegistryCI currently only supports two of them.

Git or Pkg server Tarball has been extracted Supported by RegistryCI JULIA_PKG_SERVER JULIA_PKG_UNPACK_REGISTRY
Git Yes ENV["JULIA_PKG_SERVER"] = "" delete!(ENV, "JULIA_PKG_UNPACK_REGISTRY")
Pkg server Yes Yes delete!(ENV, "JULIA_PKG_SERVER") ENV["JULIA_PKG_UNPACK_REGISTRY"] = "true"
Pkg server No No delete!(ENV, "JULIA_PKG_SERVER") delete!(ENV, "JULIA_PKG_UNPACK_REGISTRY")

To switch your registry format:

  1. import Pkg
  2. Pkg.Registry.rm("General")
  3. Run the approprate commands to set/unset the environment variables shown in the table above.
  4. Pkg.Registry.add("General")

For example, to switch to the Pkg server extracted registry:

import Pkg

Pkg.Registry.rm("General")

delete!(ENV, "JULIA_PKG_SERVER")
ENV["JULIA_PKG_UNPACK_REGISTRY"] = "true"

Pkg.Registry.add("General")

@DilumAluthge
Copy link
Member

@CameronBieganek Can you confirm that switching to one of the two supported registry formats fixes the problem for you? If so, then we can change this issue to be a feature request for RegistryCI to support the non-extracted Pkg server registry. But let's first make sure you can get things working.

@CameronBieganek
Copy link
Author

Are there any performance advantages to sticking with the unextracted tarball? If so, I'd rather stick with that. Although I suppose I could probably just delete the registry and reinstall it if I want to go back to an unextracted tarball. This isn't an important issue for me---I just wanted to check out the string distance for a package name.

@DilumAluthge
Copy link
Member

On most systems, it is more performant to use the unextracted tarball.

You could just clone the General registry to a temp directory, and pass that directory as the first positional argument to get_all_non_jll_package_names.

@DilumAluthge DilumAluthge changed the title AutoMerge.get_all_non_jll_package_names no longer works Feature request: support the compressed (non-extracted) version of the Pkg server registry Feb 28, 2022
@CameronBieganek
Copy link
Author

Cool, thanks! I'll try that when I get a chance.

@ericphanson
Copy link
Member

I think the way to do this without needing an unpacked registry would be for someone to help setup https://github.com/GunnarFarneback/RegistryInstances.jl as a stable face in front of Pkg's registry-handling internals on Julia 1.7+, and then use it here.

@GunnarFarneback
Copy link
Contributor

That package is stuck because I haven't needed it badly enough myself yet. The major missing part is to extract the relevant tests from Pkg and if necessary add some new ones.

czimm79 added a commit to czimm79/RegistryCI.jl that referenced this issue Aug 24, 2022
@czimm79
docs - default path requires an extracted registry
8312d7b 
Added a brief note explaining that `get_all_non_jll_package_names` requires a non-default extracted General registry, as explained in JuliaRegistries#442 .
@czimm79 czimm79 mentioned this issue Aug 24, 2022
Merged
bors bot added a commit that referenced this issue Aug 24, 2022
@bors @czimm79
Merge #472
0281bbf 
472: checking name similarity requires an extracted registry r=DilumAluthge a=czimm79

Added a brief note explaining that `get_all_non_jll_package_names` requires a non-default extracted General registry, as explained in #442 .

Co-authored-by: Coy Zimmermann <[email protected]>
DilumAluthge pushed a commit to czimm79/RegistryCI.jl that referenced this issue Aug 26, 2022
@czimm79 @DilumAluthge
docs - default path requires an extracted registry
2b54b98 
Added a brief note explaining that `get_all_non_jll_package_names` requires a non-default extracted General registry, as explained in JuliaRegistries#442 .
bors bot added a commit that referenced this issue Aug 26, 2022
@bors @thomvet @czimm79
Merge #470 #472
b6ffd4c 
470: Documentation: fix a typo in `private-registries.md` r=DilumAluthge a=thomvet

Minor typo.

472: checking name similarity requires an extracted registry r=DilumAluthge a=czimm79

Added a brief note explaining that `get_all_non_jll_package_names` requires a non-default extracted General registry, as explained in #442 .

Co-authored-by: Thomas Vetter <[email protected]>
Co-authored-by: Coy Zimmermann <[email protected]>
@cgarling
Copy link

cgarling commented Jul 7, 2024

Just adding that I hit this today and would be interested in this feature. Thanks for your work on the CI infrastructure!

@ericphanson ericphanson mentioned this issue Jul 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

support RegistryInstance in get_all_non_jll_package_names JuliaRegistries/RegistryCI.jl
5 participants
@GunnarFarneback @DilumAluthge @ericphanson @CameronBieganek @cgarling