Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix cross build for various python packages #369631

Merged
merged 4 commits into from
Jan 2, 2025
Merged

Conversation

DavHau
Copy link
Member

@DavHau DavHau commented Dec 31, 2024

  • python3Packages.contourpy: fix cross build
  • python3Packages.home-assistant-chip-wheels: simplify dependency closure generation
  • python3Packages.home-assistant-chip-wheels: fix cross build
  • python3Packages.bcrypt: fix cross build

Apart from the cross fixes, this contains an optimization for the dependency closure computation in home-assistant-chip-wheels.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@ofborg ofborg bot added the 6.topic: cross-compilation Building packages on a different platform than they will be used on label Dec 31, 2024
@ofborg ofborg bot requested review from Mic92, fabaff, domenkozar and mweinelt December 31, 2024 12:47
@mweinelt mweinelt requested a review from symphorien December 31, 2024 12:51
Copy link
Member

@mweinelt mweinelt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

  • The changes to bcrypt look fine
  • The changes to contourpy need work, they break the test suite and Python 3.13 support
  • For the home-assistant-chip-wheels changes I'd like @symphorien to also take a look

All changes except for chip-wheels need to pass through staging, because they're too stronk.

Copy link
Member

@symphorien symphorien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regresses #355051

@symphorien
Copy link
Member

I'm surprised you could cross compile chip at all. The list of python dependencies mixes build and runtime dependencies. Also TIL about genricClosure, thanks.

@DavHau DavHau changed the base branch from master to staging January 1, 2025 05:16
@DavHau
Copy link
Member Author

DavHau commented Jan 1, 2025

This regresses #355051

Fixed

I'm surprised you could cross compile chip at all. The list of python dependencies mixes build and runtime dependencies. Also TIL about genricClosure, thanks.

Hm, not sure. At least it does build now. We can still push more fixes later in case it doesn't work in practice. I don't use matter as of now.

Another weird thing is, when evaluating chip for a cross system, it takes very long and uses around 50GB or ram. I had to enable zram in order to build it. I wonder if such performance impact is expected because of cross, or if this has to do with this manual dependency juggling going on here.
I was hoping introducing genericClosure would fix the performance here, but it only improved it a little bit.

@symphorien
Copy link
Member

About the egregious memory use when cross building: I commented out all attributes of stdenv.mkDerivation except name and version and got the same issue, so the memory issue does not come from the computation you replaced by genericClosure.

@mweinelt
Copy link
Member

mweinelt commented Jan 1, 2025

The cause of the excessive memory usage for cross python is known, but I have no idea how to resolve this.
#338231 (comment)

@wegank wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Jan 1, 2025
@DavHau DavHau requested a review from mweinelt January 2, 2025 03:50
@mweinelt mweinelt merged commit 1660c7f into NixOS:staging Jan 2, 2025
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: cross-compilation Building packages on a different platform than they will be used on 6.topic: python 10.rebuild-darwin: 5001+ 10.rebuild-linux: 5001+ 12.approvals: 1 This PR was reviewed and approved by one reputable person
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants