clean: Optimize clean with multiple -p specifiers

[osiewicz]

Co-authored-by: dino [email protected]

What does this PR try to resolve?

This commit optimizes implementation of cargo clean -p by reducing the amount of directory walks that take place.
We now batch calls to rm_rf_prefix_list, thus potentially avoiding multiple walks over a single subdirectory. In practice this helps us significantly reduce the runtime for clearing large workspaces (as implemented in #16263); for Zed, cargo clean --workspace went down from 73 seconds to 3 seconds.

We have 216 workspace members.

How to test and review this PR?

We've tested it by hand, running it against regex, ruff and zed codebases.

This PR is still marked as draft, as I don't love the code. I would also understand if y'all were against merging this, given that new build directory layout is in flight.

[epage]

Have you tried out the performance of cargo clean --workspace with -Zbuild-dir-new-layout? Unsure what the timeline will be for stabilizing that so I can understand if you want to optimize this in the mean time.

[epage]

Are we really benefiting from all of this filesystem globbing?

Should we just walk all of the content upfront, get it into a Vec, and then do a retain call on that, pattern matching against the files for the different packages, and then delete what is left?

[osiewicz]

Have you tried out the performance of cargo clean --workspace with -Zbuild-dir-new-layout?

No, I did not try -Zbuild-dir-new-layout. I'll do that and report back.

Are we really benefiting from all of this filesystem globbing?

I don't think we benefit much from globbing, particularly because most of it's use cases are about a simple prefix/suffix matching. I'm in the process of coalescing file walking further. I'll push that up to this branch before marking this PR as ready to review.

Should we just walk all of the content upfront, get it into a Vec, and then do a retain call on that, pattern matching against the files for the different packages, and then delete what is left?

If we could make the pattern matching part straightforward (to read/maintain) then sure, that'd be best. I'm clinging onto the existing setup for no good reason; the end goal for me is to have a faster cargo clean -p --workspace.

[osiewicz]

I am very happy to report that -Zbuild-dir-new-layout would make cargo clean --workspace take about a second on Zed codebase. :)

This PR is currently at 2.6s. I think we could reach a similar ballpark with the current build dir layout.. But obviously, this code will be removed at some point either way.

[rustbot]

☔ The latest upstream changes (possibly 2b48142) made this pull request unmergeable. Please resolve the merge conflicts.