[rush] non-deterministic installs

[benkeen]

Summary

We're getting non-deterministic builds on our CI.

We have a multi-stage build where Azure ensures the exact commit hash so separate stages always get exactly the same source. We have a preliminary stage doing a rush build of our library packages which populates the remote cache, then ~10 secondary stages that repeat that step, followed by whatever they're building (tests, linting etc).

Originally we could rely on the second runs of the rush builds would pull 100% from cache, but now cache misses are extremely common. I haven't been able to detect a pattern.

Repro steps

I don't have a reproduction besides our own setup. But for MS employees, please reach out to me at [email protected] and I'd be happy to get you access to see the problem.

Details

We recently updated pnpm to v9 and Rush to 5.149.1. However, this was actually issue we're seen from the very introduction of using Rush some 2 years ago. It used to be extremely rare, now it's very common. The problem may well be with pnpm, not Rush, but I'm not sure how to debug.

Standard questions

Question	Answer
@microsoft/rush globally installed version?	5.149.1
rushVersion from rush.json?	5.149.1
useWorkspaces from rush.json?	false
Operating system?	Linux
Would you consider contributing a PR?	Yes
Node.js version (node -v)?	v18.20.4