[do not merge] Move std_detect into stdlib

[Kobzol]

This PR moves the std_detect crate from stdarch to be a part of rust-lang/rust instead.

The first commit actually moves the whole directory from the stdarch Josh subtree, so that git blame history is kept intact. Then I had to make a few changes to appease tidy.

The most complex thing here is porting the tests. We can't have std_detect both in r-l/r and stdarch, because they could get desynchronized, so we have to perform the move more or less "atomically", which means that we also have to port all the existing std_detect tests from the stdarch repository.

The stdarch repo runs the following std_detect tests:

Build

The build-std-detect.sh script (https://github.com/rust-lang/stdarch/blob/e2b6512aed87df45294ae680181eeef7a802cd95/ci/build-std-detect.sh) builds std_detect using the nightly compiler for several targets. This will be subsumed by normal x build library on our Tier 1/2 targets. However, the stdarch repository also tests the following targets:

• aarch64-unknown-freebsd
• armv6-unknown-freebsd
• powerpc-unknown-freebsd
• powerpc64-unknown-freebsd
• aarch64-unknown-openbsd

Which we don't build/test on our CI currently. I think we have mostly two options here:

1. Ignore these targets
2. Create a special CI job that will build stage 1 rustc and then cross-compile std (or just the std_detect crate?) for these targets.

Documentation

The dox.sh script (https://github.com/rust-lang/stdarch/blob/3fec5adcd52a815f227805d4959a25b6402c7fd5/ci/dox.sh) builds and documents std_detect for several targets. All of them are Tier 2/we have dist- jobs for them, so I think that we can just skip this and let our normal CI subsume it?

Tests

The run.sh script (https://github.com/rust-lang/stdarch/blob/1b201cec2cca7465602a65ed6ae60517224b15f3/ci/run.sh) runs cargo test on std_detect with a bunch of variations of feature flags. This will be subsumed by x test library in our CI. The only problem is that stdarch runs these tests for a ludicrous number of targets:

        - tuple: i686-unknown-linux-gnu
        - tuple: x86_64-unknown-linux-gnu
        - tuple: arm-unknown-linux-gnueabihf
        - tuple: armv7-unknown-linux-gnueabihf
        - tuple: aarch64-unknown-linux-gnu
        - tuple: aarch64_be-unknown-linux-gnu
        - tuple: riscv32gc-unknown-linux-gnu
        - tuple: riscv64gc-unknown-linux-gnu
        - tuple: powerpc-unknown-linux-gnu
        - tuple: powerpc64-unknown-linux-gnu
        - tuple: powerpc64le-unknown-linux-gnu
        - tuple: s390x-unknown-linux-gnu
        - tuple: i586-unknown-linux-gnu
        - tuple: nvptx64-nvidia-cuda
        - tuple: thumbv6m-none-eabi
        - tuple: thumbv7m-none-eabi
        - tuple: thumbv7em-none-eabi
        - tuple: thumbv7em-none-eabihf
        - tuple: loongarch64-unknown-linux-gnu
        - tuple: wasm32-wasip1
        - tuple: x86_64-apple-darwin
        - tuple: x86_64-apple-ios-macabi
        - tuple: aarch64-apple-darwin
        - tuple: aarch64-apple-ios-macabi
        - tuple: x86_64-pc-windows-msvc
        - tuple: i686-pc-windows-msvc
        - tuple: aarch64-pc-windows-msvc
        - tuple: x86_64-pc-windows-gnu
        - tuple: aarch64-unknown-linux-gnu
        - tuple: aarch64_be-unknown-linux-gnu
        - tuple: armv7-unknown-linux-gnueabihf
        - tuple: loongarch64-unknown-linux-gnu
        - tuple: powerpc-unknown-linux-gnu
        - tuple: powerpc64-unknown-linux-gnu
        - tuple: powerpc64le-unknown-linux-gnu
        - tuple: riscv32gc-unknown-linux-gnu
        - tuple: riscv64gc-unknown-linux-gnu
        - tuple: s390x-unknown-linux-gnu
        - tuple: x86_64-unknown-linux-gnu
        - tuple: aarch64-apple-darwin
        - tuple: aarch64-apple-ios-macabi

We definitely do not run tests for all of these targets on our CI. So maybe include them into the special job that will cross-compile std_detect for Free/OpenBSD?

r? @ghost

[rustbot]

These commits modify the library/Cargo.lock file. Unintentional changes to library/Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

There are changes to the tidy tool.

cc @jieyouxu

[Kobzol]

This is not yet ready for review, but it will need some input from stdarch maintainers regarding the test changes, as they have to happen in this PR, otherwise we either lose testing for std_detect for some time, or we will have two duplicated sources for std_detect, neither is great. Not sure who to ping exactly about the CI changes though.

CC @rust-lang/libs

[cuviper]

One of my peers is looking at implementing aarch64 outlined atomics in compiler-builtins, which needs to check for the LSE feature. Will it still be possible for it to use is_aarch64_feature_detected! if this is moved in-tree?

(That said, I'm not sure if that can work either way since the feature caching also uses atomics, so that might create a chicken-and-egg situation...)

[Kobzol]

How would it work before? std_detect wasn't published to crates.io in ~6 years. So I assume that the code just accessed std_detect through std? That should still continue working.

[tgross35]

One of my peers is looking at implementing aarch64 outlined atomics in compiler-builtins, which needs to check for the LSE feature. Will it still be possible for it to use is_aarch64_feature_detected! if this is moved in-tree?

Do you mean LSE would need to be checked within compiler-builtins? I don't think that is easily possible since AIUI std_detect for aarch64 requires filesystem access and builtins has to be dependency-free. It might be possible to do some things with weak symbols or a separate .o that gets linked only if std is available, but that gets tricky. We do some detection on x86, which is a clone of core_detect, but that doesn't require access to privileged registers https://github.com/rust-lang/compiler-builtins/blob/8aba4c899ee89eef7fe688cdfa6629ddd56908f9/libm/src/math/arch/x86/detect.rs.

(Amanieu would know more if you want to zulip this)

[cuviper]

How would it work before? std_detect wasn't published to crates.io in ~6 years. So I assume that the code just accessed std_detect through std?

The code is currently not doing this, only implementing the more basic LL/LC mode that doesn't need feature checks. That kind of defeats the point of "outlined atomics", but it's functional.

https://github.com/rust-lang/compiler-builtins/blob/8aba4c899ee89eef7fe688cdfa6629ddd56908f9/compiler-builtins/src/aarch64_linux.rs

Do you mean LSE would need to be checked within compiler-builtins?

Yes, I think so, but I'm open to suggestions.

I don't think that is easily possible since AIUI std_detect for aarch64 requires filesystem access and builtins has to be dependency-free.

Note that compiler-builtins recommends to just use compiler-rt if you care, i.e. setting bootstrap build.optimized-compiler-builtins, but that implementation is also not dependency-free. On Linux, it uses getauxval in a constructor here:

https://github.com/llvm/llvm-project/blob/6a993264ee0105da32a6a57fb077796076cf6bf4/compiler-rt/lib/builtins/cpu_model/aarch64.c#L49-L51
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/builtins/cpu_model/aarch64/lse_atomics/getauxval.inc

[cuviper]

Anyway, as I said, I'm not even sure it's possible for that to work, given that std_detect caching also uses atomics. I just wanted to raise this possibility in case it affects our choices here, to move or not.

[tgross35]

Rather than checking auxvec in compiler-builtins itself, could we check a weakly defined symbol? Then if std is available it provides this symbol, and if it's not available then compiler-builtins will fall back to the default implementation. That keeps the dependency tree clean and all of std (mostly fs and alloc) can be used to implement the feature detection. I think the proposed way of doing feature detection in core would work along those lines rust-lang/rfcs#3469 (comment).

LLVM indeed puts everything into compiler-rt, but I don't believe we need to follow exact suit here (#138944 is similar).

[cuviper]

If we allow a constructor like LLVM is doing, then we could have a static symbol like __aarch64_have_lse_atomics (not necessarily that same name) in compiler-builtins that's initialized false, and a constructor in std that checks for LSE to set it true.