Enable delta downloads of mods

[GeckoEidechse]

First up, I'm gonna point out that this would (to my knowledge) be huge undertaking as it will require a complete overhaul of the whole infrastructure, that being said:

I'd love to see some support for delta downloads. The idea basically is that when a mod-manager like r2mm updates a mod, it has the option to request only the files that changed between versions.

This way updating a huge mod, let's say >1GB where the only change is a single small file (<10MB), means that only that file needs to be downloaded from Thunderstore. This severely cuts down in download size and subsequently wait time for downloading.

The most prominent example for this that comes to my mind is Flatpak. An example run of flatpak update looks like

        ID                            Branch          Op         Remote          Download
 1. [✓] org.telegram.desktop          stable          u          flathub         48,4 MB / 75,8 MB

Notice that even though a full update would have been 75MB, it only had to download 48MB, resulting in saving around a third of the download size. From personal experience in using Flatpak this is actually a pretty bad example, I usually see a lot better ratios, with e.g. 10KB downloaded of a total download of 100MB.

For Thunderstore this could be implemented on either a per file basis, so that if a file is changed between updates, that whole file is downloaded, or even better would be downloading only the diff between the files. So if in e.g. some config file a line is changed, only that change is downloaded and then applied.

Now for this to work, either way Thunderstore would actually have to extract ZIP files and inspect them, hence requiring a pretty big rewrite of the backend.

That being said if the decision is made to implement delta downloads/updates, it could help with severely cutting down network bandwidth (and even storage if only deltas are stored).

[MythicManiac]

Hey, we actually have plans for something along these lines, but as you mentioned, it's a pretty significant undertaking to do at this point.

The primary issue isn't even overhauling anything internally, but rather how there are many 3rd party tools which expect to use the current API. So as migrating the ecosystem as a whole towards something like this is quite difficult, we've been working on the CLI based mod manager with the intention of making that the easy upgrade path for any existing tooling. Essentially the idea is that the CLI should function as an abstraction layer & programmatic interface for anything to do with mod management, which would naturally include any future changes to file downloads. Once the CLI based mod management is stable enough, we can relatively easily make changes to the ecosystem that would now be hard to do without breaking backwards compatibility.

That all said, we're likely going to implement & expose a file based API way before actually migrating everything to use it by default, and while I can't promise any timeline, I hope we'd have enough time for that during this year.

Specifically I've been looking into building a content addressable CDN for Thunderstore, as that is a much better paradigm for open ecosystems as it has less points of failure. There's some very preliminary code regarding that already here: https://github.com/thunderstore-io/ipfs-cid and the current intention is to start implementing new features with that paradigm already, and then over time migrate older features over once things seem stable.

The first implementation of the content addressable structure is most likely going to be a schema of the games supported in the ecosystem & their configs etc, as those need to be able to link to media files e.g. for the game icon, and using content IDs is a much better way of doing so than URLs which depend on specific domains.