Integrated Distributed ThinLTO (DTLTO): Design Overview

[bd1976bris]

Initial DTLTO Support

This PR introduces initial support for Integrated Distributed ThinLTO (DTLTO). DTLTO was previously discussed in an RFC and during the LTO roundtable discussion at the October US 2024 LLVM conference. PlayStation has offered this feature as a proprietary technology for some time, and we would like to see support in LLVM.

Overview of DTLTO

DTLTO enables the distribution of backend ThinLTO compilations via external distribution systems, such as Incredibuild. Existing support for distributing ThinLTO compilations typically involves separate thin-link (--thinlto-index-only), backend compilation, and link steps coordinated by a modern build system, like Bazel. This "Bazel-style" distributed ThinLTO requires a modern build system as it must handle the dynamic dependencies specified in the summary index file shards. However, adopting a modern build system can be prohibitive for users with established build infrastructure.

In contrast, DTLTO manages distribution within LLVM during the traditional link step. This approach means that DTLTO is usable with any build process that supports in-process ThinLTO.

Documentation and Resources

• RFC: Integrated Distributed ThinLTO RFC
• Documentation:
  • Feature documentation added in this PR at llvm/docs/dtlto.rst.
  • Clang and LLD CLI documentation added at clang/docs/dtlto.rst and lld/docs/dtlto.rst.
• Example Distributors: Included distributors for LIT testing added under llvm/utils/dtlto.
• Real Distribution System Support: Example scripts which work with real distribution systems are available for Windows and Linux.

Features of This Initial Commit

This commit provides a minimal but functional implementation of DTLTO, which will be expanded in subsequent commits. The current implementation includes:

• COFF and ELF support.
• Support for bitcode members in thin archives.
• Basic support for distributing backend ThinLTO compilations.
• A JSON interface that allows new distribution systems to be supported without modifying LLVM.

The goal of this initial commit is to demonstrate what will be required to support DTLTO while providing useful minimal functionality. Hopefully, having a concrete PR will facilitate discussion and review of the feature.

Performance

We have access to a large farm of computers on Windows. For a link of clang.exe on a modest Windows development machine (AMD64 16 cores, 64GB RAM) DTLTO (via sn-dbs.py) was approximately 4 times as fast as multi-threaded in-process ThinLTO.

To estimate the overhead from DTLTO vs in-process ThinLTO, we measured the difference in the time taken to link Clang with in-process ThinLTO using one thread per core, and DTLTO using one local process per core. On both Windows and Linux the overhead was approximately 6%.

Note that, to facilitate review, this PR elides performance optimizations where possible.

Planned Future Enhancements

The following features will be addressed in future commits:

• Support for the ThinLTO cache.
• Support for (non-thin) archives/libraries containing bitcode members.
• Support for more LTO configuration states e.g., basic block sections.
• Performance improvements. For example, improving the performance of the temporary file removal.

Discussion Points

• Feature Name: The DTLTO name could potentially cause confusion with the existing Bazel-style distributed ThinLTO. At the LLVM roundtable discussion no one objected to the name, but we remain open to suggestions.
• Backend Compilation Configuration: Currently, Clang is invoked to do the backend compilations and a minimal number of options are added to the Clang command line to ensure that the codegen is reasonable (for the testing we have done so far). However, it would be good to find a scalable solution for matching the code-generation state in the invoked external tool to the code-generation state if an in-process ThinLTO backend was in use.
• Clang Error Handling: There is some precedent for compiler drivers handling options that only apply to specific linkers. Should Clang emit an error if DTLTO options are used and the linker isn't LLD?

Other approaches

We have experimented with other approaches for implementing DTLTO. In particular we have explored:

• Not using a new ThinLTO backend.
• Various ways to handle (non-thin) archives.
• Use of dynamic library plugins instead of processes.

We have prepared another branch to demonstrate some of these ideas: integrated-DTLTO-no-backend

List of Child PRs:

(I intend to update this section as new PRs are filed.)

• ✅ Core LLVM functionality – PR [DTLTO][LLVM] Integrated Distributed ThinLTO (DTLTO) #127749 (merged)
• 🔄 ELF LLD support – PR [DTLTO][LLD][ELF] Add support for Integrated Distributed ThinLTO #142757 (in review)

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[bd1976bris]

⚠️ C/C++ code formatter, clang-format found issues in your code. ⚠️

You can test this locally with the following command:
View the diff from clang-format here.

I believe this failure is OK as I have followed the (non-standard) formatting in the flagged file which the code for the other ThinLTO backends use.

[romanova-ekaterina]

Thanks for the heads up, so I should not do a detailed code review for PR127749? Is there more info on what you mean by a "no-backend DTLTO"?

Actually, please review whatever you would like to at this point, Theresa. I don't want to get in the way of hearing what you think - we're keen to your input. I just wanted to point out that since another branch is coming, you may wish to wait until it arrives if you think a side-by-side comparison would be a good way of doing things. To clarify: that other branch won't be put up as a pull request, but we can decide how to proceed here if the general design shown in that other branch is preferred. I also mentioned that it will appear in a few days, but that's really dependent on the results of some more internal review. We're working hard on it!

[romanova-ekaterina]

In a couple of days, we will submit an alternative PR with "no DTLTO backend" implementation. Though from the first sight it seems logical to use a separate DTLTO backend, we thought that "no DTLTO backend" implementation will be simpler, cleaner and more importantly, will leave a solid foundation for future important performance enhancements that are planning to add in the future. These performance enhancements will be impossible to implement within DTLTO backend.

Actually, we don't have any data to determine whether or not these ideas for performance enhancements translate into real world gains. So, let's perhaps revisit these in subsequent PRs, assuming we get something landed for starters, here.

[bd1976bris]

Hi Reviewers! Thanks for the feedback here. I wanted to draw attention to something that I mentioned briefly in the description - the possibility of a plugin interface as opposed to invoking an external process that consumes JSON.

There are some theoretical advantages with plugins. For example, if distribution systems exist/arise that allow data to be passed back and forth between LLVM and a distribution system using memory buffers instead of files, a plugin could perhaps do that more efficiently. But we haven't done anything yet to quantify how much better this would be vs implicitly leaning on e.g. memory mapped files and the OS file cache. The distribution systems we're motivated to support from customer demand don't have such capabilities at this time.

Does anyone have any opinions on this?

[bd1976bris]

Reviewers have suggested that smaller PRs be created to facilitate detailed reviews while keeping this one around to provide an overall picture and facilitate discussion of high-level design (see this ongoing discussion). I have created PR #127749 to collect detailed review comments for the parts related to LLVM and have added those who commented here as reviewers.
I will do my best to keep this PR up to date with suggestions from "child" PRs such as PR #127749.

Note that I have changed the name of this PR to indicate that it provides an overview of DTLTO to avoid confusion.

Thanks again for all your input!

[romanova-ekaterina]

Hi Reviewers! Thanks for the feedback here. I wanted to draw attention to something that I mentioned briefly in the description - the possibility of a plugin interface as opposed to invoking an external process that consumes JSON.

There are some theoretical advantages with plugins. For example, if distribution systems exist/arise that allow data to be passed back and forth between LLVM and a distribution system using memory buffers instead of files, a plugin could perhaps do that more efficiently. But we haven't done anything yet to quantify how much better this would be vs implicitly leaning on e.g. memory mapped files and the OS file cache. The distribution systems we're motivated to support from customer demand don't have such capabilities at this time.

Does anyone have any opinions on this?

The “No backend DTLTO” branch is ready. Please have a look and let us know what you think. https://github.com/romanova-ekaterina/llvm-project/pull/new/kromanova/main/integrated-DTLTO-no-backend

This comment #127749 (comment) has more details about the differences between “Out of process (DTLTO) backend” branch and “No backend” DTLTO branch.

[romanova-ekaterina]

Thanks for the heads up, so I should not do a detailed code review for PR127749? Is there more info on what you mean by a "no-backend DTLTO"?

Actually, please review whatever you would like to at this point, Theresa. I don't want to get in the way of hearing what you think - we're keen to your input. I just wanted to point out that since another branch is coming, you may wish to wait until it arrives if you think a side-by-side comparison would be a good way of doing things. To clarify: that other branch won't be put up as a pull request, but we can decide how to proceed here if the general design shown in that other branch is preferred. I also mentioned that it will appear in a few days, but that's really dependent on the results of some more internal review. We're working hard on it!

The “No backend DTLTO” branch is ready. Please have a look and let us know what you think. https://github.com/romanova-ekaterina/llvm-project/pull/new/kromanova/main/integrated-DTLTO-no-backend

This comment #127749 (comment) has more details about the differences between “Out of process (DTLTO) backend” branch and “No backend” DTLTO branch.

[bd1976bris]

Thanks to everyone for the review effort on DTLTO — I’ve now merged the LLVM portion of the changes (#127749).

I’m currently on PTO this week, but once I’m back, I’ll post the Clang and LLD patches for review. In the meantime, if anyone wants to try out DTLTO on a stable branch, I’ve backported the changes to LLVM 19 here: https://github.com/bd1976bris/llvm-project/commits/dtlto_llvm19.

[bd1976bris]

I have put a PR for introducing support for Integrated Distributed ThinLTO (DTLTO) in ELF LLD: #142757. Once that's merged I will follow up with PRs for adding DTLTO support to Clang and adding DTLTO support to COFF LLD.