Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add initial support for delta updates #308

Merged
merged 37 commits into from
Jan 5, 2025
Merged

feat: add initial support for delta updates #308

merged 37 commits into from
Jan 5, 2025

Conversation

R1kaB3rN
Copy link
Member

@R1kaB3rN R1kaB3rN commented Dec 10, 2024 •
edited
Loading

Related to #133 (comment) and #134.

Adds optional support for delta updates for currently supported compatibility tools, resulting in significantly faster updates to the latest releases. This also puts umu-launcher up to parity with the Steam client in terms of update distribution for its tools.

For clients, this can be enabled by setting either PROTONPATH=GE-Latest or PROTONPATH=UMU-Latest where each of those values represent the latest GE-Proton and UMU-Proton respectively. In this system, those builds will be placed within the private directory $XDG_DATA_HOME/umu/compatibilitytools, and like Steam's Proton Experimental, only 1 build will be maintained. For this system to work, it will be expected for clients or users to not modify the contents of those builds.

Delta updates to the Steam Linux Runtime will be supported later after we start building it, and will become the default. Additionally, this feature is currently not supported for zipapps.

For this feature, cbor2, pyzstd and xxhash will be a new optional dependencies. Additionally, cargo will be a new build dependency.

TODO:

  • Refactor code to pull patch file from Github or OWC controlled server

@R1kaB3rN R1kaB3rN force-pushed the delta branch 2 times, most recently from 5b6b1cc to dfd85aa Compare December 24, 2024 19:37
@R1kaB3rN R1kaB3rN marked this pull request as ready for review December 24, 2024 19:47
@R1kaB3rN
Copy link
Member Author

Will resolve the Debian build failure in a bit and write the basic tests later. For testing, this suffices for now.

@R1kaB3rN
Copy link
Member Author

R1kaB3rN commented Dec 24, 2024 •
edited
Loading

@beh-10257 I'm interested to know if the build succeeds on NixOS. Do you mind testing how this builds on NixOS? I'd appreciate it.

Also, if you're interested in testing this, I'll attach the patch file intended to upgrade GE-Proton9-20 below in a bit. To test, first install cbor2 then install/copy GE-Proton9-20 to $HOME/.local/share/umu/compatibilitytools then rename it to GE-Latest. If you have it installed already and are copying it over, be sure to delete all the __pycache__.

Afterwards, assuming you have the patch in your home directory, run:
PROTONPATH=GE-Latest GAMEID=umu-0 UMU_DELTA=$HOME/delta.cbor umu-run ""

@R1kaB3rN
Copy link
Member Author

https://drive.proton.me/urls/18EM1D76FG#VlasLi0bh7Qa

@beh-10257
Copy link
Contributor

ok will do will build this branch thanks also can I send a patch that adds cbor2 as optional dependency
(to make my life easier now)

@R1kaB3rN
Copy link
Member Author

ok will do will build this branch thanks also can I send a patch that adds cbor2 as optional dependency (to make my life easier now)

It's fine right now, because cbor2 is already an optional dependency.

See

umu-launcher/pyproject.toml

Line 45 in dfd85aa

updates = ["cbor2>=5.4.6,<6.0.0"]

@R1kaB3rN
Copy link
Member Author

R1kaB3rN commented Dec 24, 2024 •
edited
Loading

In other words, it will not be installed by default for the system package umu-launcher. Though, we'll probably need to make it clearer in the Nix flake. Because cbor2 will be needed to activate delta updates

@R1kaB3rN
Copy link
Member Author

In the Flake, the latest stable version of Rust will need to be added as a build dependency too. Perhaps that's called pkgs.rustc?

@beh-10257
Copy link
Contributor

I mean I added cbor2 as an optional dependency I mean I can turn it off by default but why not
heres a patch for cbor2
wait I can't upload .patch files ???

@beh-10257
Copy link
Contributor

In the Flake, the latest stable version of Rust will need to be added as a build dependency too. Perhaps that's called pkgs.rustc?

as an optional or a main dependency

@beh-10257
Copy link
Contributor

0001-add-cbor2-as-an-optional-dependency-for-nixos.txt

just rename .txt to .patch

@beh-10257
Copy link
Contributor

beh-10257 commented Dec 24, 2024 •
edited
Loading

merge that patch into the branch
I will test the delta updates in around 30 minutes
I'm downloading the cbor2 file right now
also
image
I have this can't I copy ge-proton-9-20 and see cbor actually do its magic
or will it have to download from scratch to do its thing
also what is ??
__pycache__
aka where is the location

@R1kaB3rN
Copy link
Member Author

In the Flake, the latest stable version of Rust will need to be added as a build dependency too. Perhaps that's called pkgs.rustc?

as an optional or a main dependency

I meant Rust as a build dependency in depsBuildBuild.

umu-launcher/packaging/nix/umu-launcher.nix

Line 8 in 6189d0d

depsBuildBuild = [

@beh-10257
Copy link
Contributor

feel free to add rustc (on a sidenote) cargo is not included so if its being used add it as well

@R1kaB3rN
Copy link
Member Author

R1kaB3rN commented Dec 24, 2024 •
edited
Loading

merge that patch into the branch I will test the delta updates in around 30 minutes I'm downloading the cbor2 file right now also image I have this can't I copy ge-proton-9-20 and see cbor actually do its magic or will it have to download from scratch to do its thing also what is ?? __pycache__ aka where is the location

The python interpreter in the runtime creates a __pycache__ folder that stores its compiled bytecode. It does this so it'll run faster for future launches, and it's also part of the reason why launching a new downloaded build can be a bit slow. In GE-Proton9-20, you should see folders like __pycache__.

@R1kaB3rN
Copy link
Member Author

If you have the fd program, within GE-Proton9-20, you can run the command fd '__pycache__' . -X rm -r to get rid of it.

@beh-10257
Copy link
Contributor

image
I mean there is nothing

@R1kaB3rN
Copy link
Member Author

R1kaB3rN commented Dec 24, 2024 •
edited
Loading

feel free to add rustc (on a sidenote) cargo is not included so if its being used add it as well

Yeah, cargo is needed. Not just rustc, and I can add that.

@beh-10257
Copy link
Contributor

after searching for pycache in my entire home steam is the only program that actually creates those folders
no mention of umu so we are good to go I guess

@R1kaB3rN
Copy link
Member Author

image I mean there is nothing

Strange, unless you're using bwrap that binds that path as read-only? If that's the case then the __pycache__ won't persist and you'll be regenerating it each launch

@R1kaB3rN
Copy link
Member Author

after searching for pycache in my entire home steam is the only program that actually creates those folders no mention of umu so we are good to go I guess

Alright. Yeah, so if there's not cache go ahead and copy GE-Proton9-20 to $HOME/.local/share/umu/compatibilitytools then rename it to GE-Latest.

@beh-10257
Copy link
Contributor

image I mean there is nothing

Strange, unless you're using bwrap that binds that path as read-only? If that's the case then the __pycache__ won't persist and you'll be regenerating it each launch

what is read only ??
the proton directory in the bwrap of umu ??

@beh-10257
Copy link
Contributor

after searching for pycache in my entire home steam is the only program that actually creates those folders no mention of umu so we are good to go I guess

Alright. Yeah, so if there's not cache go ahead and copy GE-Proton9-20 to $HOME/.local/share/umu/compatibilitytools then rename it to GE-Latest.

ok will do (I'm waiting for you to merge that patch tbh)
and also redownloading ge proton 9-21 to see if pycache shows up

@R1kaB3rN
Copy link
Member Author

image I mean there is nothing

Strange, unless you're using bwrap that binds that path as read-only? If that's the case then the __pycache__ won't persist and you'll be regenerating it each launch

what is read only ?? the proton directory in the bwrap of umu ??

Yeah, that's what I meant. Since I know you use bwrap, unless you're not doing anything special with the proton.

@R1kaB3rN
Copy link
Member Author

Should be good now. I applied your patch and added cargo as a build dependency.

@beh-10257
Copy link
Contributor

PROTONPATH=GE-Latest GAMEID=umu-0 UMU_DELTA=$HOME/delta.cbor umu-run ""

do I need the umu_delta bit since I have ge proton 20

@beh-10257
Copy link
Contributor

update it doesn't build since cargo access the internet
and thats impure

is cargo only used for bzip2

R1kaB3rN and others added 24 commits January 5, 2025 02:25
@R1kaB3rN
umu_proton: add function to fetch patch
b9b3a3c
@R1kaB3rN
umu_proton: support finding delta builds
554f660
@R1kaB3rN
umu_proton: remove redundant install_proton
25c73d2
@R1kaB3rN
umu_proton: refactor install for delta builds
57fdae6
@R1kaB3rN
umu_proton: refactor pre-install steps for delta builds
25d272a
@R1kaB3rN
umu_proton: add functionality to apply delta updates
0b0cedd
@R1kaB3rN
umu_test: update tests
5f77789
@R1kaB3rN
umu_test: add initial delta update tests
878e8e1
@R1kaB3rN
deb: update deb package
9e025d5
@R1kaB3rN
rpm: add optional dependencies
cd86930
@R1kaB3rN
snap: add optional dependencies
14d9ca2
@R1kaB3rN @LovingMelody @beh-10257
nix: update flake
4d42d33 
Co-authored-by: Melody <[email protected]>
Co-authored-by: beh_10257 <[email protected]>
@R1kaB3rN
ci: add new dependencies to workflows
3ce5b65
@R1kaB3rN @beh-10257
README: update description for nix
ab75b81 
Co-authored-by: beh_10257 <[email protected]>
@R1kaB3rN
.gitignore: update ignore
dc6b04b
@R1kaB3rN
docs: update description for PROTONPATH
f6d1dbf
@R1kaB3rN
pyproject: fix parsing error
a0d11f4
@R1kaB3rN
umu_proton: fix log statement
1464afa 
- Wrong Proton build was printed to stderr
@R1kaB3rN
umu_proton: update style
18a15e4
@R1kaB3rN
umu_proton: fetch patch from endpoint
37408ec
@R1kaB3rN
umu_proton: implement _fetch_patch
2c39efc
@R1kaB3rN
umu_proton: add GE-Latest codename to set
2835802
@R1kaB3rN
umu_proton: fix not waiting for future
205bca6
@R1kaB3rN
umu_proton: fix mypy lint
bc7f83e 
- Ignore the type. There's an open issue upstream for this, and we can refactor for readability later if desired
@R1kaB3rN
umu_proton: fix mypy lint
cda115b
@R1kaB3rN
ci: don't use latest runner
28d3d69 
- For now, run the e2e tests on Ubuntu 22.04 because the tests are failing due to not using the bwrap apparmor profile:

pressure-vessel-wrap[2993]: E: Child process exited with code 1: bwrap: setting up uid map: Permission denied

Fix this later in a separate commit or pull request
@R1kaB3rN R1kaB3rN merged commit 313c82a into Open-Wine-Components:main Jan 5, 2025
12 checks passed
@R1kaB3rN R1kaB3rN deleted the delta branch January 5, 2025 21:23
@R1kaB3rN R1kaB3rN mentioned this pull request Jan 8, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LovingMelody LovingMelody LovingMelody approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@R1kaB3rN @beh-10257 @LovingMelody