Enable MIR optimizations in test suite again

[RalfJung]

miri's test suite used to also run with enabled MIR optimizations, but does not any more because of too many bugs. We should get it enabled again, eventually.

• For run-pass, see rustup #926 (comment)
• For compile-fail, the current blocker is MIR inlining does not handle SourceScopeLocalData rust#66137.

[RalfJung]

The run-pass test suite works with optimizations now, but the compile-fail test suite still explodes with higher optimization levels.

[RalfJung]

Had to disable optimizations for run-pass again, likely caused by rust-lang/rust#63580.

[wesleywiser]

I think I will have some time to look at this later tonight or tomorrow morning.

[wesleywiser]

So, I'm trying to test some local changes to rustc I have and I'm getting build errors when I run ./miri test:

error[E0463]: can't find crate for `core`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `unicode-width`.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `term`.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `proc_macro`.

To learn more, run the command again with --verbose.
error: `"/home/wesley/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo" "build" "--release" "--manifest-path" "/tmp/xargo.uuhYMKXriUiG/Cargo.toml" "--target" "x86_64-unknown-linux-gnu" "-p" "test"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace
fatal error: Failed to run xargo

I've built my local rustc with ./x.py build --stage 2, I've added it to rustup as a custom toolchain, and I've run rustup override in the miri repo. Any ideas what might be causing that?

[RalfJung]

That is strange. I regularly build Miri against a local rustc. Usually I do ./x.py build src/rustc, maybe that makes the difference? The directory you added as custom toolchain is .../build/x86_64-unknown-linux-gnu/stage2 (maybe with a different target triple), right? Which OS/platform are you on?

In which part of the Miri build process does the error occur? Is it while building Miri, or while building the custom libstd? Please post a complete build log, starting with your ./miri invocation.

[wesleywiser]

Ok, so here's what I just tried:

1. In ~/code/rust/rust run ./x.py build src/rustc
2. In ~/code/rust/miri run rustup override set stage2 which links to ~/code/rust/rust/build/x86_64-unknown-linux-gnu/stage2
3. In ~/code/rust/miri run ./miri test

Which does this:

wesley@endurance:~/code/rust/miri> ./miri test
   Compiling autocfg v0.1.6
   Compiling proc-macro2 v0.4.30
   Compiling unicode-xid v0.1.0
   Compiling libc v0.2.62
   Compiling cc v1.0.41
   Compiling syn v0.15.44
   Compiling proc-macro2 v1.0.1
   Compiling unicode-xid v0.2.0
   Compiling failure_derive v0.1.5
   Compiling syn v1.0.5
   Compiling rustc-demangle v0.1.16
   Compiling memchr v2.2.1
   Compiling bitflags v1.1.0
   Compiling cfg-if v0.1.9
   Compiling serde v1.0.99
   Compiling ryu v1.0.0
   Compiling log v0.4.8
   Compiling rand_core v0.4.2
   Compiling lazy_static v1.4.0
   Compiling getrandom v0.1.11
   Compiling byteorder v1.3.2
   Compiling itoa v0.4.4
   Compiling regex-syntax v0.6.11
   Compiling quick-error v1.2.2
   Compiling semver-parser v0.7.0
   Compiling termcolor v1.0.5
   Compiling hex v0.3.2
   Compiling rustc-workspace-hack v1.0.0
   Compiling shell-escape v0.1.4
   Compiling thread_local v0.3.6
   Compiling rand_core v0.3.1
   Compiling rand_jitter v0.1.4
   Compiling humantime v1.2.0
   Compiling rand_hc v0.1.0
   Compiling rand_xorshift v0.1.1
   Compiling rand_isaac v0.1.1
   Compiling num-traits v0.2.8
   Compiling num-integer v0.1.41
   Compiling rand_chacha v0.1.1
   Compiling rand_pcg v0.1.2
   Compiling rand v0.6.5
   Compiling aho-corasick v0.7.6
   Compiling quote v0.6.13
   Compiling quote v1.0.2
   Compiling time v0.1.42
   Compiling dirs-sys v0.3.4
   Compiling atty v0.2.13
   Compiling rand_os v0.1.3
   Compiling directories v2.0.2
   Compiling regex v1.2.1
   Compiling chrono v0.4.7
   Compiling backtrace-sys v0.1.31
   Compiling env_logger v0.6.2
   Compiling vergen v3.0.4
   Compiling synstructure v0.10.2
   Compiling backtrace v0.3.35
   Compiling serde_derive v1.0.99
   Compiling failure v0.1.5
   Compiling miri v0.1.0 (/home/wesley/code/rust/miri)
   Compiling semver v0.9.0
   Compiling serde_json v1.0.40
   Compiling rustc_version v0.2.3
   Compiling cargo_metadata v0.8.2
    Finished release [optimized] target(s) in 1m 14s
    Finished release [optimized] target(s) in 0.02s
     Running `target/release/cargo-miri miri setup`
    Updating crates.io index
   Compiling compiler_builtins v0.1.19
   Compiling core v0.0.0 (/home/wesley/code/rust/rust/src/libcore)
   Compiling cc v1.0.41
   Compiling libc v0.2.62
   Compiling build_helper v0.1.0 (/home/wesley/code/rust/rust/src/build_helper)
   Compiling cmake v0.1.42
   Compiling unwind v0.0.0 (/home/wesley/code/rust/rust/src/libunwind)
   Compiling rustc_tsan v0.0.0 (/home/wesley/code/rust/rust/src/librustc_tsan)
   Compiling rustc_asan v0.0.0 (/home/wesley/code/rust/rust/src/librustc_asan)
   Compiling rustc_lsan v0.0.0 (/home/wesley/code/rust/rust/src/librustc_lsan)
   Compiling rustc_msan v0.0.0 (/home/wesley/code/rust/rust/src/librustc_msan)
   Compiling std v0.0.0 (/home/wesley/code/rust/rust/src/libstd)
   Compiling rustc-std-workspace-core v1.0.0 (/home/wesley/code/rust/rust/src/tools/rustc-std-workspace-core)
   Compiling cfg-if v0.1.8
   Compiling alloc v0.0.0 (/home/wesley/code/rust/rust/src/liballoc)
   Compiling panic_abort v0.0.0 (/home/wesley/code/rust/rust/src/libpanic_abort)
   Compiling rustc-std-workspace-alloc v1.0.0 (/home/wesley/code/rust/rust/src/tools/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/home/wesley/code/rust/rust/src/libpanic_unwind)
   Compiling hashbrown v0.5.0
    Finished release [optimized] target(s) in 28.99s
   Compiling unicode-width v0.1.6
   Compiling term v0.0.0 (/home/wesley/code/rust/rust/src/libterm)
   Compiling proc_macro v0.0.0 (/home/wesley/code/rust/rust/src/libproc_macro)
error[E0463]: can't find crate for `core`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `unicode-width`.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `proc_macro`.
warning: build failed, waiting for other jobs to finish...
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `term`.

To learn more, run the command again with --verbose.
error: `"/home/wesley/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo" "build" "--release" "--manifest-path" "/tmp/xargo.kHu0e3jhZph2/Cargo.toml" "--target" "x86_64-unknown-linux-gnu" "-p" "test"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace
fatal error: Failed to run xargo
wesley@endurance:~/code/rust/miri> 

[RalfJung]

That looks like your xargo is outdated. What does xargo --version say? Are you sure you are using the latest master of Miri?

[wesleywiser]

Are you using the latest master of miri?

Yes, I just rebased my changes on master (3a5c44c2d83b04651a2e04178087e23e3db6fccc).

That looks like your xargo is outdated.

I don't remember ever installing xargo so I assume ./miri did that?

> xargo --version
xargo 0.3.16
cargo 1.38.0-nightly (677a180f4 2019-07-08)

[wesleywiser]

This is very strange. If I use the latest nightly (2019-08-31), ./miri test works fine. Furthermore, if I re-enable the hashmap tests, they also pass under the latest nightly. So I guess this is fixed?

I'll submit a PR to re-enable the tests and verify it works on the CI.

Edit: I just realized the tests are passing on macOS on CI and I'm currently using my MacBook. I'll retest this on Linux.

Edit 2: I am seeing the test failure on Linux. I think perhaps I'm missing some kind of configuration necessary to use miri with a local rustc build. If I use a nightly, then miri compiles fine but if I use my local copy (with up to date HEAD and xargo), then I get the above errors.

[RalfJung]

./miri test does not use the bad opt-level any more. I didn't want to hold up Miri on this issue so I reduced the level we are using for testing.

Try this:

RUST_BACKTRACE=1 ./miri run-debug tests/run-pass/hashmap.rs -Zmir-opt-level=2

[wesleywiser]

Thanks, that repro's on nightly.

I tried running various parts of the ./miri script by hand and I noticed that if I run cargo run --release --bin cargo-miri -- miri setup on my stage2 toolchain, I get this:

    Finished release [optimized] target(s) in 0.07s
     Running `target/release/cargo-miri miri setup`
Installing rust-src component: `rustup component add rust-src`
error: toolchain 'stage2' does not support components
info: caused by: invalid toolchain name: 'stage2'
fatal error: Failed to install rust-src component

Should it be trying to install the rust-src component? Do I need to do something special to get that component to point to my local repo?

[RalfJung]

You need to set XARGO_RUST_SRC to the src dir of your Rust toolchain. Components don't work with linked toolchains.

Also, the HashMap test indeed takes a somewhat different code path on macOS due to #686, which might explain the platform differences.