-
-
Notifications
You must be signed in to change notification settings - Fork 15.1k
bug: LLVM22 upgrade causes severe compile time regression #157302
Copy link
Copy link
Open
Labels
A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.C-bugCategory: This is a bug.Category: This is a bug.I-compiletimeIssue: Problems and improvements with respect to compile times.Issue: Problems and improvements with respect to compile times.I-prioritizeIssue: Indicates that prioritization has been requested for this issue.Issue: Indicates that prioritization has been requested for this issue.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.Relevant to the compiler team, which will review and decide on the PR/issue.needs-triageThis issue may need triage. Remove it if it has been sufficiently triaged.This issue may need triage. Remove it if it has been sufficiently triaged.regression-untriagedUntriaged performance or correctness regression.Untriaged performance or correctness regression.
Metadata
Metadata
Assignees
Labels
A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.C-bugCategory: This is a bug.Category: This is a bug.I-compiletimeIssue: Problems and improvements with respect to compile times.Issue: Problems and improvements with respect to compile times.I-prioritizeIssue: Indicates that prioritization has been requested for this issue.Issue: Indicates that prioritization has been requested for this issue.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.Relevant to the compiler team, which will review and decide on the PR/issue.needs-triageThis issue may need triage. Remove it if it has been sufficiently triaged.This issue may need triage. Remove it if it has been sufficiently triaged.regression-untriagedUntriaged performance or correctness regression.Untriaged performance or correctness regression.
Type
Fields
Give feedbackNo fields configured for issues without a type.
During work on https://github.com/zama-ai/kms we noticed that upgrading the toolchain from 1.94 to 1.95 caused compile times to balloon from 12min to 2h30m. We set it aside and then tried again with 1.96: same result.
The investigation shows that compilation gets stuck on one of our dependencies,
tfhe-fft. The problem is specific to Linux/x86, where the AVX-512/pulpcodegen path causes a ~45x slowdown; running with--timingsconfirms that this is a codegen issue, so distinct from the Bevy's issue with the frontend.Looking at the compilation with
perf tophints at the SLP vectorizer, but that's where our investigation ended.Code
This is the most minimal repro we've been able to cook up. Run
cargo new slp-reproand replaceCargo.tomlwith:Leave
src/main.rsas the generated stub. Then:Important
tfhe-fft'savx512feature →pulp/x86-v4). On aarch64 that path is scalar and there is no regression.cargo build), not--release. A--releasebuild is fast (because it strips the optimize-with-debug-info combination?); the regression needstfhe-fftatopt-level = 3under the dev profile.Version it regressed in
First shipped in stable 1.95.0 (the first release with LLVM 22); also present in 1.96.0 and every nightly since.
Regression bisection
Bisected with
cargo-bisect-rustc(predicate = "tfhe-fft build exceeds 600s"):That commit is #150722 ("Update to LLVM 22",
rustc/21.1→rustc/22.1-2026-01-27) — a pure LLVM submodule bump, no rustc codegen changes.the SLP vectorizer?
Toggling only
-C llvm-args=-vectorize-slp=false, with everything else fixed, isolates the cost to the SLP vectorizer. Coldcargo buildoftfhe-fft, wall-clock:-vectorize-slp=false)So ≈99% of the regression is the SLP vectorizer. With SLP disabled, the LLVM-22 build (199s) is faster than the LLVM-21 build (286s).