select need be able to recognise exec_compatible_with and target_compatible_with

[xiedeacc]

Description of the feature request:

my project import rules_proto and grpc, when I cross compile for arm, rules_proto will choose a wrong platform protoc, host platform protoc expected but arm platform protoc choosed, but I think it's not rules_proto's bug, for:

my platforms value pass to bazel was

platform(
    name = "linux_gcc11_aarch64_none_musleabi",
    constraint_values = [
        "@platforms//cpu:aarch64",
        "@platforms//os:linux",
        ":gcc_11",
        ":musleabi",
    ],
)

toolchain defined by myself:

toolchain(
    name = "gcc11_arm_aarch64_none_gnueabi_xcompile_toolchain",
    exec_compatible_with = [
        "@platforms//cpu:x86_64",
        "@platforms//os:linux",
    ],
    target_compatible_with = [
        "@platforms//cpu:aarch64",
        "@platforms//os:linux",
        "//platforms:gcc_11",
        "//platforms:gnueabi",
    ],
    toolchain = "@gcc11_arm_aarch64_none_gnueabi//:cc_toolchain",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

rules_proto use select to decide use which platform protoc, cpu:x86_64 and os:linux from target_compatible_with will be recognised by rules_proto, so linux-aarch64 choosed, it's a wrong choice, it didn't recognise it's a cross compile. so I think there exists two way to solve this problem:

1. rules_proto choose protoc by detect host platform, not by select
2. make bazel compitable with this situation, I think select should be able to recogise exec_compatible_with and target_compatible_with

alias(
    name = "protoc",
    actual = select({
        ":linux-aarch64": "@com_google_protobuf_protoc_linux_aarch64//:protoc",
        ":linux-ppc": "@com_google_protobuf_protoc_linux_ppc//:protoc",
        ":linux-s390x": "@com_google_protobuf_protoc_linux_s390_64//:protoc",
        ":linux-x86_32": "@com_google_protobuf_protoc_linux_x86_32//:protoc",
        ":linux-x86_64": "@com_google_protobuf_protoc_linux_x86_64//:protoc",
        ":macos-aarch64": "@com_google_protobuf_protoc_macos_aarch64//:protoc",
        ":macos-x86_64": "@com_google_protobuf_protoc_macos_x86_64//:protoc",
        ":windows-x86_32": "@com_google_protobuf_protoc_windows_x86_32//:protoc",
        ":windows-x86_64": "@com_google_protobuf_protoc_windows_x86_64//:protoc",
        "//conditions:default": "@com_github_protocolbuffers_protobuf//:protoc",
    }),
    visibility = ["//visibility:public"],
)

[xiedeacc]

bazelbuild/rules_proto#126

[meteorcloudy]

@sgowroji This issue belongs to the configurability team

[katre]

Can I reproduce this on a linux device? I don't have an arm device handy.

Can you provide a minimal reproduction repository with your toolchain available?

To clarify, the issue is that you are building the protoc compiler from source, and it's being built for the target platform but then run on the execution platform, and that fails because you are cross-compiling?

[xiedeacc]

yes you're right, you can reproduce on a linux device

[katre]

If you have a small reproduction repo that I can clone that will help diagnose the issue.

[katre]

Unfortunately I can't work on this further without being able to reproduce it. I'm going to lower the priority and mark this as triaged until you can get back to us.

[sgowroji]

@xiedeacc, We are marking the above Issue as stale because it has not had any recent activity from many days.
It will be closed in 7 days if there is no further activity occurs. Thank you.

[sgowroji]

Closing as stale. Please reopen if you'd like to work on this further.

[cristifalcas]

I would like to help with this, since we are affected as well.

I use the precompiled @com_google_protobuf//:protoc binary, which unfortunately resolves to the target platform and not the exec platform.

What is the expectation for this (I'm on mac):

BUILD.bazel:

platform(
    name = "linux_aarch64",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:aarch64",
    ],
)

platform(
    name = "macos_x86_64",
    constraint_values = [
        "@platforms//os:macos",
        "@platforms//cpu:x86_64",
    ],
)

Bazel command:

$ bazel run @com_google_protobuf//:protoc --platforms //:linux_aarch64 --extra_execution_platforms //:macos_x86_64
....
/bin/bash: /private/var/tmp/_bazel_cfalcas/856dc74a4318774eeb1496e4c7291b94/execroot/com_etsy_search/bazel-out/darwin-fastbuild/bin/external/com_google_protobuf_protoc_linux_aarch64/protoc.exe: cannot execute binary file

Same output for

$ bazel run @com_google_protobuf//:protoc --platforms //:linux_aarch64

I'm expecting to run the protoc binary for the exec platform

[cristifalcas]

I believe the actual select is here

[katre]

I'm not sure bazel run interacts properly with the execution/target platform (ideally, bazel run wouldn't let you set --platforms, and would instead build for the host platform, since that's where the command will execute). I see that the protoc used is actually protoc.exe, are you running bazel on a Windows device?

Is this also a bug when protoc is invoked as part of an action? I'll re-open and take a look at this again, but I don't think the original request (select checks exec_compatible_with) is actually part of the problem.

[katre]

Also note that today is my last working day of the year, so I won't be looking at this again until January.

[cristifalcas]

no hurry, we just started working towards cross compiling

I believe it's the same issue, because it's the same select as the OP, which returns the target platform.

I'll make a repo with a rule to show the issue.

About the exe in the name, that is artificially added, the binary is linux_aarch64:

$ bazel run @com_google_protobuf//:protoc --platforms //tools:linux_aarch64 --extra_execution_platforms //tools:macos_x86_64 --sandbox_debug
.....
/bin/bash: /private/var/tmp/_bazel_cfalcas/856dc74a4318774eeb1496e4c7291b94/execroot/com_etsy_search/bazel-out/darwin-fastbuild/bin/external/com_google_protobuf_protoc_linux_aarch64/protoc.exe: cannot execute binary file

$ file /private/var/tmp/_bazel_cfalcas/856dc74a4318774eeb1496e4c7291b94/execroot/com_etsy_search/bazel-out/darwin-fastbuild/bin/external/com_google_protobuf_protoc_linux_aarch64/protoc.exe
/private/var/tmp/_bazel_cfalcas/856dc74a4318774eeb1496e4c7291b94/execroot/com_etsy_search/bazel-out/darwin-fastbuild/bin/external/com_google_protobuf_protoc_linux_aarch64/protoc.exe: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=67bf1d2acddf41eab6f53c16a746eb8c4e681526, stripped

Notice the ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1

[katre]

Right, but the Windows subsystem that adds .exe isn't platform aware, which is why I was curious.

[cristifalcas]

Ok, I found my "issue" and it's an issue at all, I was using the wrong config from rules_rust.
I was hitting the problem that was solved in this PR and I had to change, in my case, in crates_repository rule the attribute build_script_data to build_script_tools.

And the exe suffix is appended by protocolbuffers for all artefacts

Sorry for the noise

[katre]

Thanks for following up, I think this can be closed now. Feel free to comment or open a new issue as needed.