Cannot cross-compile Rust target from Apple M1 -> x86_64-linux-gnu

[huyquanha]

Describe the bug

I'm following the example here with some changes in order to compile a Rust binary (I'm using Apple M1 macbook) for a Docker image (linux-amd64/x86_64). rules_docker image transition is enabled to select the correct target platform.

However, the compilation fail with:

error[E0514]: found crate `async_stream_impl` compiled by an incompatible version of rustc
   --> external/crate_index__async-stream-0.3.4/src/lib.rs:238:13
    |
238 |     pub use async_stream_impl::{stream_inner, try_stream_inner};
    |             ^^^^^^^^^^^^^^^^^
    |
    = note: the following crate versions were found:
            crate `async_stream_impl` compiled by rustc 1.67.1 (d5a82bbd2 2023-02-07): /private/var/tmp/_bazel_kevinh/18f8c22285dcb10eba63dccc5c60f984/execroot/__main__/bazel-out/darwin_arm64-opt-exec-C7777A24-ST-53d1b78224ec/bin/external/crate_index__async-stream-impl-0.3.4/libasync_stream_impl-2887306267.dylib
    = help: please recompile that crate using this compiler (rustc 1.67.1 (d5a82bbd2 2023-02-07) (built from a source tarball)) (consider running `cargo clean` first)

I'm not even sure why but the 2 compiler versions are exactly the same. if I changed the version of RUST_VERSION in my WORKSPACE to something else, like 1.68.2 I still got the same error, but this time one of the 2 compiler versions display 1.68.2 instead.

Is this somehow an issue with rust toolchain having one version and Nix toolchain with another version of the compiler and the 2 conflicts with each other?

To Reproduce
My WORKSPACE is almost exactly the same as in the provided Rust example. The major difference is I add exec_constraints and target_constraints to both nixpkgs_cc_configure and nixpkgs_rust_configure in order to target linux-x86_64.

Here's the relevant section in WORKSPACE to set up Nix + Rust:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# Load platforms explicitly because rules_nixpkgs seems to have an outdated platforms dependency
# that doesn't satisfy rules_rust.
http_archive(
    name = "platforms",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/platforms/releases/download/0.0.6/platforms-0.0.6.tar.gz",
        "https://github.com/bazelbuild/platforms/releases/download/0.0.6/platforms-0.0.6.tar.gz",
    ],
    sha256 = "5308fc1d8865406a49427ba24a9ab53087f17f5266a7aabbfc28823f3916e1ca",
)

RUST_EDITION = "2021"

http_archive(
    name = "rules_rust",
    sha256 = "2466e5b2514772e84f9009010797b9cd4b51c1e6445bbd5b5e24848d90e6fb2e",
    urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.18.0/rules_rust-v0.18.0.tar.gz"],
    patches = [
        "//:patches/001-rules-rust-crate-universe.patch",
    ],
)

http_archive(
    name = "io_tweag_rules_nixpkgs",
    sha256 = "b01f170580f646ee3cde1ea4c117d00e561afaf3c59eda604cf09194a824ff10",
    strip_prefix = "rules_nixpkgs-0.9.0",
    urls = ["https://github.com/tweag/rules_nixpkgs/archive/refs/tags/v0.9.0.tar.gz"],
)

load("@io_tweag_rules_nixpkgs//nixpkgs:repositories.bzl", "rules_nixpkgs_dependencies")

rules_nixpkgs_dependencies()

load("@io_tweag_rules_nixpkgs//nixpkgs:nixpkgs.bzl", "nixpkgs_cc_configure", "nixpkgs_local_repository")

# I use `niv` for nixkpgs version management, hence these file deps
nixpkgs_local_repository(
    name = "nixpkgs",
    nix_file = "//:nixpkgs.nix",
    nix_file_deps = [
        "//:nix/sources.nix",
        "//:nix/sources.json",
    ],
)

# rules_rust also uses the cc compiler.
nixpkgs_cc_configure(
    name = "nixpkgs_config_cc",
    exec_constraints = [
        "@platforms//os:osx",
        "@platforms//cpu:arm64",
    ],
    nix_file_deps = [
        "//:nix/sources.nix",
        "//:nix/sources.json",
    ],
    repository = "@nixpkgs",
    target_constraints = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
)

load("@io_tweag_rules_nixpkgs//nixpkgs:toolchains/rust.bzl", "nixpkgs_rust_configure")

nixpkgs_rust_configure(
    name = "nix_rust",
    default_edition = RUST_EDITION,
    exec_constraints = [
        "@platforms//os:osx",
        "@platforms//cpu:arm64",
    ],
    nix_file_deps = [
        "//:nix/sources.nix",
        "//:nix/sources.json",
    ],
    repository = "@nixpkgs",
    target_constraints = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
)

load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")

RUST_VERSION = "1.67.1"
rules_rust_dependencies()

# To register toolchains to support different platforms.
rust_register_toolchains(
    edition = RUST_EDITION,
    extra_target_triples = ["x86_64-unknown-linux-gnu"],
    versions = [
        RUST_VERSION,
    ],
)

# https://bazelbuild.github.io/rules_rust/crate_universe.html
load("@rules_rust//crate_universe:repositories.bzl", "crate_universe_dependencies")

crate_universe_dependencies(
    rust_version = RUST_VERSION,
    bootstrap = True,
    rust_toolchain_cargo_template = "@nix_rust//:bin/{tool}",
    rust_toolchain_rustc_template = "@nix_rust//:bin/{tool}",
)

load("@rules_rust//crate_universe:defs.bzl", "crate", "crates_repository", "render_config")

crates_repository(
    name = "crate_index",
    cargo_lockfile = "//:Cargo.lock",
    # The generated lock file by Bazel
    lockfile = "//:cargo-bazel-lock.json",

    packages = {
        ...,
        "async-stream": crate.spec(
            version = "0.2",
        ),
        ...
    },
    generator = "@cargo_bazel_bootstrap//:cargo-bazel",
    render_config = render_config(
        default_package_name = "",
    ),
    rust_toolchain_cargo_template = "@nix_rust//:bin/{tool}",
    rust_toolchain_rustc_template = "@nix_rust//:bin/{tool}",
)

load("@crate_index//:defs.bzl", "crate_repositories")

crate_repositories()

My .bazelrc is:

build --host_platform=@io_tweag_rules_nixpkgs//nixpkgs/platforms:host
build --crosstool_top=@nixpkgs_config_cc//:toolchain

I use rule rust_image from rules_docker which references another rust_binary target, but when I try to run bazel run ${my_rust_image_target} I got the above error.

Expected behavior
Cross compilation should work from Apple M1 execution platform -> linux-amd64 as a target platform.

Environment

• OS name + version: Ventura 13.2.1, M1
• Version of the code: rules_nixpkgs 0.9.0

[benradf]

From a cursory glance this looks to be a similar issue to #303 (though with Rust rather than C++). Essentially, the default rustc binary provided by nixpkgs is not a cross-compiler and cannot be used to compile Linux targets on MacOS.

The references to build triples and constraints are a little misleading here. They influence Bazel during toolchain resolution, but do not make the underlying rustc binary that Nix provides into a cross-compiler. For that you would need to provide a custom Nix expression that produces a cross-compiler. We recognise this isn't ideal and #347 aims to improve the situation.

[huyquanha]

Thanks @benradf, that makes sense!

Is there anyway I can unblock myself while waiting for that? Not really a Nix expert myself, I'm not sure how to change nixpkgs.nix in the Rust example to provide a rustc cross-compiler

let
  spec = builtins.fromJSON (builtins.readFile ./nixpkgs.json);
  nixpkgs = fetchTarball {
    url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
    sha256 = spec.sha256;
  };
in
import nixpkgs

My nixpkgs.nix is a bit different cuz I use niv for nixpkgs version management

let sources = import ./nix/sources.nix;
in
import sources.nixpkgs

The link above seems to be more how to set up shell.nix to expose a cross compiler to nix shell. I'm not sure if that's applicable here or nixpkgs.nix has to be modified somehow to take effect on nixpkgs_rust_configure

[benradf]

So I haven't had time to try this properly but I think if you change:

pkgs = import <nixpkgs> {{ config = {{}}; overrides = []; }};

to

pkgs = (import <nixpkgs> {{ config = {{}}; overrides = []; }}).pkgsCross.gnu64;

here in _rust_nix_contents, you should get cross versions of the packages (including rustc).

Of course, you can't directly change _rust_nix_contents as it's in rules_nixpkgs, but you can provide a nix_file_content parameter to nixpkgs_rust_configure. You will need to copy the whole _rust_nix_contents nix expression and modify the pkgs = ... line to use the cross packages.

Note that this will break non-cross-compilation scenarios, but if you can get cross-compilation working first we can then look at doing it in a way that supports both.

PS: Before making any Bazel changes you should probably see if you can get Nix to build you a cross-compiler:

nix-build -E '
  let
    pkgs = import <nixpkgs> {
      config = { };
      overrides = [ ];
    };
  in pkgs.pkgsCross.gnu64.rustc
'

[benradf]

Closing this as it's a specific instance of the more general Better API for cross-compilation support issue.