Skip to content

bug: LLVM22 upgrade causes severe compile time regression #157302

@dvdplm

Description

@dvdplm

During work on https://github.com/zama-ai/kms we noticed that upgrading the toolchain from 1.94 to 1.95 caused compile times to balloon from 12min to 2h30m. We set it aside and then tried again with 1.96: same result.

The investigation shows that compilation gets stuck on one of our dependencies, tfhe-fft . The problem is specific to Linux/x86, where the AVX-512/pulp codegen path causes a ~45x slowdown; running with --timings confirms that this is a codegen issue, so distinct from the Bevy's issue with the frontend.

Looking at the compilation with perf top hints at the SLP vectorizer, but that's where our investigation ended.

Code

This is the most minimal repro we've been able to cook up. Run cargo new slp-repro and replace Cargo.toml with:

[package]
name = "slp-repro"
version = "0.1.0"
edition = "2021"

[dependencies]
tfhe-fft = { version = "=0.10.1", features = ["fft128", "serde"] }

# tfhe-fft must be optimized for the SLP blowup to appear. Keep the dev profile
# (debug-info + assertions on); a `--release` build does NOT reproduce.
[profile.dev]
opt-level = 3

Leave src/main.rs as the generated stub. Then:

$ cargo +1.94.0 build      # LLVM 21 — finishes in ~5 minutes
$ cargo +1.96.0 build      # LLVM 22 — takes hours

Important

  • x86_64 only. The blowup is in the AVX-512 path (tfhe-fft's avx512 feature → pulp/x86-v4). On aarch64 that path is scalar and there is no regression.
  • Build in dev (cargo build), not --release. A --release build is fast (because it strips the optimize-with-debug-info combination?); the regression needs tfhe-fft at opt-level = 3 under the dev profile.

Version it regressed in

First shipped in stable 1.95.0 (the first release with LLVM 22); also present in 1.96.0 and every nightly since.

1.94.0 (LLVM 21): good
1.95.0 (LLVM 22): bad   (untested directly, but contains the regressing commit)
1.96.0 (LLVM 22): bad

Regression bisection

Bisected with cargo-bisect-rustc (predicate = "tfhe-fft build exceeds 600s"):

searched nightlies: from nightly-2026-01-22 to nightly-2026-01-29
regressed nightly:  nightly-2026-01-29
regressed commit:   466ea4e6c39f8a43727edcc726ca86b499e14d83

That commit is #150722 ("Update to LLVM 22", rustc/21.1rustc/22.1-2026-01-27) — a pure LLVM submodule bump, no rustc codegen changes.

the SLP vectorizer?

Toggling only -C llvm-args=-vectorize-slp=false, with everything else fixed, isolates the cost to the SLP vectorizer. Cold cargo build of tfhe-fft, wall-clock:

LLVM 21 (1.94.0) LLVM 22 (1.96.0)
SLP on (default) 286s ~9200s
SLP off (-vectorize-slp=false) 145s 199s

So ≈99% of the regression is the SLP vectorizer. With SLP disabled, the LLVM-22 build (199s) is faster than the LLVM-21 build (286s).

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-LLVMArea: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues.C-bugCategory: This is a bug.I-compiletimeIssue: Problems and improvements with respect to compile times.I-prioritizeIssue: Indicates that prioritization has been requested for this issue.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.needs-triageThis issue may need triage. Remove it if it has been sufficiently triaged.regression-untriagedUntriaged performance or correctness regression.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions