Record failed tests with `--record`, and rerun them with `--rerun` by jdonszelmann · Pull Request #154586 · rust-lang/rust

jdonszelmann · 2026-03-30T13:17:37Z

This adds two parameters to x test:

`--record`

Writes a file, by default build/failed-tests, but this can be overwritten with

[build]
record_failed_tests_path = "somepath"

with a list of all tests that fail that run.

`--rerun`

Looks for the failed-tests file, parse it, and attempt to rerun only those tests. No cli-arguments are necessary, i.e.

x test tests/ui --record
x test --rerun

Will run all failed uitests. No need to pass tests/ui to the rerun invocation.

The last commit is a little awkward, but I think it's the best way to make it so that we first run all tests that have to be rerun, and then rerun tests passed through the cli.

This makes it so:

x test tests/ui --rerun

will first rerun failed tests, some of which may be uitests, if any fail it quits and reports failed tests, but if all pass it will run all normally passed tests. In other words, only if all previously-failed tests pass on the rerun, we then also run uitests.
Without the last commit, this would instead just run all uitests, since the failed tests form a subset of all uitests. I think that's less useful.

rustbot · 2026-03-30T13:17:41Z

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

rustbot · 2026-03-30T13:17:44Z

r? @clubby789

rustbot has assigned @clubby789.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: bootstrap
bootstrap expanded to 6 candidates
Random selection from Mark-Simulacrum, clubby789, jieyouxu

jyn514 · 2026-03-30T13:27:05Z

Writes a file, by default build/failed-tests, but this can be overwritten with

I wouldn't make this configurable.

What does x test --rerun do if there was no previous --record run? How does cache invalidation work for the failed-tests file, do you just keep it around forever?

jdonszelmann · 2026-03-30T13:30:31Z

@jyn514

What does x test --rerun do if there was no previous --record run?

It will warn, but treat it as if the file was empty.

How does cache invalidation work for the failed-tests file, do you just keep it around forever?

Keeps it around forever, or at least until you --record again. It prints that it's rerunning, and from what file so it should be easy to delete or even edit manually. Rerunning acts pretty much as if you passed the test paths on the cli, though we do run them before the cli-passed paths.

WaffleLapkin · 2026-03-30T15:44:27Z

Rerunning acts pretty much as if you passed the test paths on the cli, though we do run them before the cli-passed paths.

If the build/failed-tests is empty/non-present, does that make x test --rerun run all tests?

WaffleLapkin · 2026-03-30T16:03:30Z

This is purely a speculation I suppose, but I feel like I will often forget to use --record, before the tests already failed. Would it make sense to save failures of the last command to a separate file and allow moving them to build/failed-tests?

jyn514 · 2026-03-30T16:17:58Z

@WaffleLapkin as long as you don’t modify the compiler, compiletest will remember which tests succeeded the last time, so if you add ––failed it won’t take very long at all to regenerate that list.

jdonszelmann · 2026-03-30T16:19:42Z

I agree with jyn here, shouldn't make such a big difference

jdonszelmann · 2026-03-30T16:20:45Z

Rerunning acts pretty much as if you passed the test paths on the cli, though we do run them before the cli-passed paths.

If the build/failed-tests is empty/non-present, does that make x test --rerun run all tests?

that is true I think. What do you expect the behavior to be? maybe if no file is found, and no paths are given, exit, but if some paths are explicitly given run the explicit ones?

jdonszelmann · 2026-03-30T16:20:50Z

with a warning of course

jyn514 · 2026-03-30T16:21:44Z

I think x test --rerun with no previous --record should behave exactly the same as x test.

jdonszelmann · 2026-03-30T17:10:17Z

mhm, well that's the current behavior. Except the warning of course, that you passed --rerun with nothing to rerun. But I think that's nice

jdonszelmann · 2026-03-30T18:53:18Z

cc @jieyouxu (you self assigned the other one, that one was in preparation for this one, also I figured out that bug for this one)

clubby789 · 2026-03-31T12:18:40Z

I haven't looked at the full implementation yet, but why not always record failed tests?

jyn514 · 2026-03-31T12:32:23Z

Because a later invocation might only rerun a subset of tests. Say you have this series of invocations:

$ x test ui
# ... 34 failures in ui/linkage and ui/attributes
$ x build library # assume there was a compiler change
$ x test --rerun tests/ui/attributes
# ... all tests now pass
$ x test --rerun tests/ui/linkage
# ... no tests are run :( we expected all the failed linkage tests to rerun.

clubby789 · 2026-03-31T12:36:10Z

Sorry, always *except when using --rerun 😅

jdonszelmann · 2026-04-01T06:45:33Z

I still think it's a bad idea to always record: I want it to be explicit when this file gets overwritten. I might want to

x test tests/ui --record
x test tidy
x test --rerun

That's just an example, but in general what I want this feature to solve is that ignored tests get invalidated by rebuilding the compiler because of a change, something implicit. I want this recording to be explicit so you're in control of it. So you can even temporarily switch branches to try something out and come back and have your little set of tests that make your new feature break.

The last part, about switching branches is a little idea I had: we could at some point, not now, try to integrate this with git and see if we can detect branch changes and warn when you overwrite or something. Or even write to a different file (without checking it into git) etc. but if we do logic like that, that's for later

rustbot · 2026-04-06T08:29:52Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

jdonszelmann · 2026-04-06T08:30:16Z

@jieyouxu @clubby789 rebased on the other PR now that it's merged

jieyouxu

(Initial review pass)

View changes since this review

src/bootstrap/src/core/build_steps/test/failed_tests.rs

src/bootstrap/src/core/build_steps/test.rs

src/bootstrap/src/core/config/flags.rs

jieyouxu · 2026-04-06T12:57:52Z

src/bootstrap/src/core/build_steps/test/failed_tests.rs

+        let trimmed = line.as_str().trim();
+        let without_revision =
+            trimmed.rsplit_once("#").map(|(before, _)| before).unwrap_or(&trimmed);
+        let without_suite_prefix = without_revision
+            .strip_prefix("[")
+            .and_then(|rest| rest.split_once("]"))
+            .map(|(_, after)| after.trim())
+            .unwrap_or(without_revision);


Discussion [TEST-HARNESS-JSON 1/2]: hm I thought about this. This is kinda doing a weird thing of post-processing compiletest-emitted test names, so we introduce an implicit dependency on compiletest test name formatting, which I can't say I'm a huge fan of.

I guess this is not the end of the world...

The caveat is that bootstrap captures test harness JSON output to then post-render the outcome messages. This can be from:

libtest JSON when e.g. cargo test is used on say some library crate or such,

but it can also come from compiletest-emitted messages which are more "liberal" in how the concept of "test name" is used (that the test name can have a suite prefix or revision suffixes).

If we want to avoid taking on this implicit dependency, we probably will have to introduce separate variants for compiletest-emitted test outcome JSON versus that of libtest, which doesn't really feel that great IMO. That, or maybe we can introduce optional revision / test suite fields for the message variants (which will not be present from other libtest-emitted messages but will be for compiletest-emitted messages).

NB. this was not possible previously, because previously compiletest also shelled out to libtest test executor so compiletest don't have control over the libtest executor emitted libtest JSON message format. But now compiletest uses its own executor and emits emulated libtest JSON messages. So technically I think it might be possible to change the emitted JSON format.

cc @Zalathar (this may be of interest to you)

jieyouxu · 2026-04-06T13:08:43Z

src/bootstrap/src/core/build_steps/test/failed_tests.rs

+    const MAX_RERUN_PRINTS: usize = 10;
+
+    for line in lines {
+        let trimmed = line.as_str().trim();


Suggestion [TEST-HARNESS-JSON 2/2]: I don't feel like hard-blocking this fuctionality on [TEST-HARNESS-JSON 1/2] considerations, but can you please leave a FIXME to consider changing the compiletest-emitted JSON message formats, and maybe change the failed-tests file to have like JSON lines that looks like

{ "suite": "ui", "path": "tests/ui/foo.rs" } { "suite": "run-make", "path": "tests/run-make/hello-world/rmake.rs" }

src/bootstrap/src/core/build_steps/test/failed_tests.rs

jieyouxu · 2026-04-06T13:14:48Z

src/bootstrap/src/core/build_steps/compile.rs

        // If only `compiler` was passed, do not run this step.
        // Instead the `Assemble` step will take care of compiling Rustc.
-        if run.builder.paths == vec![PathBuf::from("compiler")] {
+        if run.builder.paths(IsForRerunningTests::DontCare) == vec![PathBuf::from("compiler")] {


Discussion:

The last commit is a little awkward, but I think it's the best way to make it so that we first run all tests that have to be rerun, and then rerun tests passed through the cli.

I really do not like this viral complexity that cross-cuts all these test steps. Can we keep the rerun functionality simple, to only rerun the set of failed tests?

hm, is it that complex? Cause I do really like doing x test --rerun tests/ui and using that to run a small part (that likely errors) first and then try everything

rustbot · 2026-04-06T13:22:00Z

Reminder, once the PR becomes ready for a review, use @rustbot ready.

jieyouxu · 2026-04-12T07:54:35Z

@rustbot review

rust-bors · 2026-04-13T22:15:04Z

☔ The latest upstream changes (presumably #155253) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Mar 30, 2026

rustbot assigned clubby789 Mar 30, 2026

This comment has been minimized.

Sign in to view

jdonszelmann force-pushed the record-rerun branch from 9bd3be5 to e5e9e3b Compare March 30, 2026 13:45

This comment has been minimized.

Sign in to view

jdonszelmann force-pushed the record-rerun branch from e5e9e3b to 28032e6 Compare March 30, 2026 14:03

This comment has been minimized.

Sign in to view

jdonszelmann force-pushed the record-rerun branch from 28032e6 to a73b6b8 Compare March 30, 2026 14:32

This comment has been minimized.

Sign in to view

jdonszelmann force-pushed the record-rerun branch from a73b6b8 to 740c441 Compare March 30, 2026 14:58

jieyouxu self-assigned this Mar 31, 2026

jdonszelmann force-pushed the record-rerun branch from 740c441 to 848602d Compare April 2, 2026 09:04

jdonszelmann added 3 commits April 6, 2026 10:25

set up for recording failed tests

a29764d

rerun failed tests

2de0c9a

rerun first, then do normal tests as passed through the cli

b6965a5

jdonszelmann force-pushed the record-rerun branch from 848602d to bb19e7d Compare April 6, 2026 08:29

This comment has been minimized.

Sign in to view

jdonszelmann force-pushed the record-rerun branch from bb19e7d to 0e47a91 Compare April 6, 2026 09:22

jieyouxu requested changes Apr 6, 2026

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 6, 2026

change tracker

71e6780

jdonszelmann force-pushed the record-rerun branch from 0e47a91 to 71e6780 Compare April 10, 2026 14:50

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 12, 2026

Uh oh!

Conversation

jdonszelmann commented Mar 30, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

--record

--rerun

Uh oh!

rustbot commented Mar 30, 2026

Uh oh!

rustbot commented Mar 30, 2026

Uh oh!

This comment has been minimized.

jyn514 commented Mar 30, 2026

Uh oh!

jdonszelmann commented Mar 30, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

WaffleLapkin commented Mar 30, 2026

Uh oh!

WaffleLapkin commented Mar 30, 2026

Uh oh!

jyn514 commented Mar 30, 2026

Uh oh!

jdonszelmann commented Mar 30, 2026

Uh oh!

jdonszelmann commented Mar 30, 2026

Uh oh!

jdonszelmann commented Mar 30, 2026

Uh oh!

jyn514 commented Mar 30, 2026

Uh oh!

jdonszelmann commented Mar 30, 2026

Uh oh!

jdonszelmann commented Mar 30, 2026

Uh oh!

clubby789 commented Mar 31, 2026

Uh oh!

jyn514 commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clubby789 commented Mar 31, 2026

Uh oh!

jdonszelmann commented Apr 1, 2026

Uh oh!

rustbot commented Apr 6, 2026

Uh oh!

jdonszelmann commented Apr 6, 2026

Uh oh!

This comment has been minimized.

jieyouxu left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jieyouxu Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jieyouxu Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jieyouxu Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

jdonszelmann Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

rustbot commented Apr 6, 2026

Uh oh!

jieyouxu commented Apr 12, 2026

Uh oh!

rust-bors bot commented Apr 13, 2026

jdonszelmann commented Mar 30, 2026 •

edited by rustbot

Loading

`--record`

`--rerun`

jyn514 commented Mar 31, 2026 •

edited

Loading

jieyouxu left a comment •

edited by rustbot

Loading

jieyouxu Apr 6, 2026 •

edited

Loading