Use compressed download of library index

[per1234]

Describe the request

Download the compressed library index archive:

https://downloads.arduino.cc/libraries/library_index.tar.bz2

(as is done by Arduino CLI)

🙂 The duration of a linting operation on a a machine with a slower Internet connection will be reduced.
🙂 Bandwidth usage will be reduced.

Describe the current behavior

Data about the libraries and releases distributed via the Arduino Library Manager system is available from an index file hosted on Arduino's download server.

Arduino Lint downloads this index file and utilizes the information in several rules related to Arduino library projects.

Due to the amazing efforts of the Arduino community in creating tens of thousands of releases of thousands of libraries, this index has grown to a significant size (47 MB).

Arduino Lint downloads the index file in an uncompressed form:

arduino-lint/internal/project/projectdata/library.go

Line 74 in 2b321d7

httpResponse, err := http.Get(librariesmanager.LibraryIndexURL.String())

https://github.com/arduino/arduino-cli/blob/1f8d0f6c0dd35a55e2ddd80df5751333377ca5e7/arduino/libraries/librariesmanager/download.go#L23

var LibraryIndexURL, _ = url.Parse("https://downloads.arduino.cc/libraries/library_index.json")

🙁 This is inefficient.

Arduino Lint version

2b321d7

Operating system

All

Operating system version

Any

Additional context

This may mitigate hangs, and eventual panics, that occasionally occur while downloading the index (the root cause being unknown problems with network infrastructure):

https://github.com/arduino/library-registry/actions/runs/20200290266/job/57991196538#step:9:27

panic: stream error: stream ID 1; INTERNAL_ERROR; received from peer

goroutine 1 [running]:
github.com/arduino/arduino-lint/internal/project/projectdata.InitializeForLibrary({0xc00031e610?, 0xc8cc60?, 0xc00031e660?})
	/home/build/internal/project/projectdata/library.go:88 +0x685
github.com/arduino/arduino-lint/internal/project/projectdata.Initialize({0xc00031e610?, 0x7f34b51c15e8?, 0xc0002459e0?})
	/home/build/internal/project/projectdata/projectdata.go:39 +0x65
github.com/arduino/arduino-lint/internal/rule.Runner({0xc00031e610?, 0xc00036e5b0?, 0x1?})
	/home/build/internal/rule/rule.go:37 +0xcb
github.com/arduino/arduino-lint/internal/command.ArduinoLint(0xc000174a00?, {0xc00036e5b0, 0x1, 0x7})
	/home/build/internal/command/command.go:77 +0x3e5
github.com/spf13/cobra.(*Command).execute(0xc000370008, {0xc000034090, 0x7, 0x7})
	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0xc000370008)
	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(0x0?)
	/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 +0x13
main.main()
	/home/build/main.go:33 +0x18

In the specific context of the "Manage PRs" workflow that runs Arduino Lint on libraries submitted to Arduino Library Registry, this has the side effect of causing a spurious error on the subsequent workflow step that is intended to consume the output from a legitimate linting failure:

Error: The template is not valid. Newtonsoft.Json.JsonReaderException: Error reading JToken from JsonReader. Path '', line 0, position 0.
   at Newtonsoft.Json.Linq.JToken.ReadFrom(JsonReader reader, JsonLoadSettings settings)
   at Newtonsoft.Json.Linq.JToken.ReadFrom(JsonReader reader)
   at GitHub.DistributedTask.Expressions2.Sdk.Functions.FromJson.EvaluateCore(EvaluationContext context, ResultMemory& resultMemory)
   at GitHub.DistributedTask.Expressions2.Sdk.ExpressionNode.Evaluate(EvaluationContext context)
   at GitHub.DistributedTask.Expressions2.Sdk.Operators.Index.EvaluateCore(EvaluationContext context, ResultMemory& resultMemory)
   at GitHub.DistributedTask.Expressions2.Sdk.ExpressionNode.Evaluate(EvaluationContext context)
   at GitHub.DistributedTask.Expressions2.Sdk.Operators.Index.EvaluateCore(EvaluationContext context, ResultMemory& resultMemory)
   at GitHub.DistributedTask.Expressions2.Sdk.ExpressionNode.Evaluate(EvaluationContext context)
   at GitHub.DistributedTask.Expressions2.Sdk.Operators.GreaterThan.EvaluateCore(EvaluationContext context, ResultMemory& resultMemory)
   at GitHub.DistributedTask.Expressions2.Sdk.ExpressionNode.Evaluate(EvaluationContext context)
   at GitHub.DistributedTask.Expressions2.Sdk.Operators.And.EvaluateCore(EvaluationContext context, ResultMemory& resultMemory)
   at GitHub.DistributedTask.Expressions2.Sdk.ExpressionNode.Evaluate(EvaluationContext context)
   at GitHub.DistributedTask.Expressions2.Sdk.ExpressionNode.GitHub.DistributedTask.Expressions2.IExpressionNode.Evaluate(ITraceWriter trace, ISecretMasker secretMasker, Object state, EvaluationOptions options)
   at GitHub.DistributedTask.ObjectTemplating.Tokens.TemplateToken.EvaluateTemplateToken(TemplateContext context, String expression, Int32& bytes)
   at GitHub.DistributedTask.ObjectTemplating.Tokens.BasicExpressionToken.EvaluateTemplateToken(TemplateContext context, Int32& bytes)
   at GitHub.DistributedTask.ObjectTemplating.TemplateUnraveler.RootBasicExpression()

Examples of such spurious failures:

• Update repositories.txt library-registry#7293 (comment)
• Add repository URL for ok_micro_dock library-registry#7306 (comment)
• Update repositories.txt library-registry#7362 (comment)
• Update repositories.txt library-registry#7408 (comment)

---

Keywords

• "gzip"

• "bzip"

• "tarball"

• "zip"

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the latest nightly build
• My request contains all necessary details