advice for recovering from corrupted depot in the manual

[tpapp]

Occasionally parts of the depot become corrupted (because of interrupted downloads/processes, bugs in Pkg, etc). While this is not part of the normal workflow, it would be great if the manual could have a section on how to recover from these situations, that could be linked eg on Discourse.

Eg

1. establish the depot path (examples below assume the default)
2. remove ~/.julia/packages
3. remove the global manifest,
4. restart julia and do a pkg> resolve?

[StefanKarpinski]

Might be worth adding a pkg> fsck command that goes through verifying things and trying to fix them and/or delete/recreate them.

[DilumAluthge]

1. Commit all changes in any packages that you have checked out for development
2. Push all changes to a remote (GitHub, GitLab, etc.)
3. rm -rf ~/.julia

And of course, if you don't have any packages checked out for development, then you simply do:

1. rm -rf ~/.julia

[giordano]

I'd never suggest to people to nuke the julia directory. Why not just move it?

[tpapp]

I agree with @giordano on this --- we should never recommend just deleting things.

Until pkg> fsck is added, it would be great to determine the gentlest minimum solution. It is needed occasionally to recover from various problems, and we should have a standard way.

My understanding is the following:

1. one can remove artifacts, clones, compiled, packages, registries, they will just be downloaded at no cost if the machine is online (we should mention this, some environments have no net connection)

2. when dev exists, it may contain uncommitted stuff, and it is generally harmless

3. logs, prefs, and config are generally not a source of inconsistent state, so they can be left alone

4. environments is really small, and should be just moved to a backup location, eg

   mv ~/.julia/environments ~/.julia/environments-backup-`date +%Y-%m-%d`

Is this correct? Did I miss anything?

[StefanKarpinski]

Can't think of anything else. However, logs can cause problems since manifest usage is tracked there and the file can become corrupted (invalid TOML due to multiple writers).

[fredrikekre]

For 1.: compiled is not downloaded but generated, safe to remove just like the others though.

[tpapp]

Thinking about this, it may be preferable to include something in Pkg instead of a shell script that people copy-paste. I am thinking of something like the following:

pkg> nuke
This command needs to be invoked with --yes to do anything; this is just a message.

It can be used to recover from an inconsistent state of the package depot by removing 
files that were downloaded or automatically generated. The removed files will be
saved to a backup directory (which is displayed).

Used with --environment, it also resets your default environment.

--yes would not autocomplete.

pkg> nuke --yes
Moving files to /home/tamas/.julia_depot_backup_2020-03-23_1042.

pkg> nuke --yes --environment
Moving files to /home/tamas/.julia_depot_backup_2020-03-23_1044, resetting environment.

[StefanKarpinski]

Was thinking of something similar myself.

[tpapp]

Since corrupted registry questions come up daily on Discourse, I am wondering how to move forward with this. In order of increasing difficulty:

1. write up a bunch of commands for users to copy/paste,
2. implement nuke above,
3. implement fsck (cf pkg> fsck #1717)

[3f6a]

one can remove artifacts, clones, compiled, packages, registries, they will just be downloaded at no cost if the machine is online (we should mention this, some environments have no net connection)

I'm not sure about registries. One can have a LocalRegistry there, with changes not synced to GitHub.

[3f6a]

I think I might have a corrupted depot now, so I'm going through this.

logs can cause problems ...

Is logs safe to delete?

There is repl_history.jl there which I wouldn't want to lose. But the other files:

artifact_usage.toml, manifest_usage.toml, orphaned.toml, scratch_usage.toml

can be safely deleted?

[3f6a]

Okay, not sure if this might be relevant here. But I tried to delete: artifacts, clones, compiled, packages. Then I reopened julia and did pkg> up, and it started generating everything but eventually got this error:

Precompiling project...
  267 dependencies successfully precompiled in 142 seconds. 8 already precompiled.
  1 dependency had warnings during precompilation:
┌ ChainRulesCore [d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4]
│  ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for CompatLinearAlgebraExt [dbe5ba0b-aecc-598a-a867-79051b540f49]
│  │   exception = ArgumentError: Invalid checksum in cache file ~/.julia/compiled/v1.9/CompatLinearAlgebraExt/Zxpzq_nrSU6.dylib.
│  └ @ Base loading.jl:1818
│  ┌ Error: Error during loading of extension CompatLinearAlgebraExt of Compat, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace: