Artifact download reuse and validation

[sstock]

Problem statement:

• Installer always downloads artifacts even when they already exist locally.
• Installer does not validate the download is intact.
• On metered connections this potentially poses an undue cost.
• On slow connections this may pose an unnecessary time lag.
• On unstable/slow connections multiple launcher restarts may be required due to download failures.

Proposed solution:

• Change fabric-meta to include binary artifact size and hash.
  • https://maven.fabricmc.net/ already contains md5, sha1, sha256, and sha512 hashes for each binary.
  • Structure should allow multiple hashes for migration to other algorithms.
  • Note: Minecraft JSON already includes size and sha1 for binaries.
  • Note: Java 8 supports all hashes present in maven.fabricmc.net repository.
• Improve download algorithm to support continuation and hash validation.
• Downloads may optionally use a shared cache location, e.g. -libraryCache=$HOME/.fabric-installer

Proposed download pattern:

• Download manifest containing library information (unchanged from current behavior).
• If no existing downloaded file, begin full download.
• If existing downloaded file is too small use HTTP Range request to download remaining bytes.
  • This is supported by maven.fabricmc.net and launcher.mojang.com
  • URLConnection can be used to perform byte range downloads, although it is a bit clunky.
• Validate downloaded file using hash.
• If size matches but hash does not, remove local file and start download again.
  • Abort after more than some small number of attempts.

This assumes the binary artifacts are not changed after publication. If it is possible for a released version to change then the hash value should be stored locally and compared with the downloaded hash from the manifest to determine if the library has changed.

My experience with Fabric is extremely limited, having learned about it less than 48 hours ago. But it does seem apparent that one goal is keeping the fabric-installer small; such as using a minimal number of dependencies. To this end I did quickly verify that URLConnection may be used to implement HTTP byte range requests. It isn't as nice as with other HTTP client libraries, such as needing to parse the HTTP status line, but it is possible. Additionally, given the target is Java 8, the newer Java 9+ HttpClient is not available.

A real-world example involving a slow connection:

$ time java -jar fabric-installer-0.7.4.jar server -mcversion 1.16.5
Loading Fabric Installer: 0.7.4
Using latest loader version
Installing Fabric Loader 0.11.6(1.16.5) on the server
Downloading required files
Downloading library net.fabricmc:tiny-mappings-parser:0.2.2.14
[removed most output -- not significant in this case]
Done, start server by running fabric-server-launch.jar

real    7m6.103s
user    0m3.120s
sys     0m0.426s

$ time java -jar fabric-installer-0.7.4.jar server -mcversion 1.16.5
[removed all output -- not significant in this case]

real    6m51.211s
user    0m3.172s
sys     0m0.467s

[modmuss50]

Adding the hashes is something that needs to be done. I will look into doing it as part of the 0.12 loader release. Outside of the server all of the downloading is left to the launcher and isnt something we can control. I dont have an exact number but I would guess loader + all the fabric libs to be well under 10mb, with the upcoming 0.12 release this should be even less.