Out of memory error while building clang

[LukasReil]

I cloned llvm (ddef380) with git and followed the build process, as detailed in the clang setup guide.
I used ninja, as detailed in Using Ninja alongside Visual Studio to build everything and then encountered the following crash:

[1916/4145] Building AMDGPUGenGlobalISel.inc...
FAILED: lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc D:/Programmieren/C++/clang_build/llvm-project/build_ninja/lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc
C:\Windows\system32\cmd.exe /C "cd /D D:\Programmieren\C++\clang_build\llvm-project\build_ninja && D:\Programmieren\C++\clang_build\llvm-project\build_ninja\bin\llvm-tblgen.exe -gen-global-isel -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU -ID:/Programmieren/C++/clang_build/llvm-project/build_ninja/include -ID:/Programmieren/C++/clang_build/llvm-project/llvm/include -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target --long-string-literals=0 D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU/AMDGPUGISel.td --write-if-changed -o lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc -d lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc.d"
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: D:\\Programmieren\\C++\\clang_build\\llvm-project\\build_ninja\\bin\\llvm-tblgen.exe -gen-global-isel -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU -ID:/Programmieren/C++/clang_build/llvm-project/build_ninja/include -ID:/Programmieren/C++/clang_build/llvm-project/llvm/include -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target --long-string-literals=0 D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU/AMDGPUGISel.td --write-if-changed -o lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc -d lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc.d
Exception Code: 0x80000003
 #0 0x00e861d9 HandleAbort D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\Support\Windows\Signals.inc:429:0
 #1 0x64342434 (C:\Windows\SYSTEM32\ucrtbased.dll+0xc2434)
 #2 0x64343a52 (C:\Windows\SYSTEM32\ucrtbased.dll+0xc3a52)
 #3 0x00e01871 llvm::report_bad_alloc_error(char const *, bool) D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\Support\ErrorHandling.cpp:177:0
 #4 0x00e0221f out_of_memory_new_handler D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\Support\ErrorHandling.cpp:190:0
 #5 0x64456388 (C:\Windows\SYSTEM32\MSVCP140D.dll+0x26388)
 #6 0x6432f962 (C:\Windows\SYSTEM32\ucrtbased.dll+0xaf962)
 #7 0x010204f7 operator new(unsigned int) D:\a\_work\1\s\src\vctools\crt\vcstartup\src\heap\new_scalar.cpp:41:0
 #8 0x008183fc std::_Default_allocate_traits::_Allocate(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:101:0
 #9 0x00809b1c std::_Allocate_manually_vector_aligned<struct std::_Default_allocate_traits>(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:158:0
#10 0x008099bf std::_Allocate<8, struct std::_Default_allocate_traits>(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:222:0
#11 0x008831e0 std::allocator<struct llvm::gi::MatchTableRecord>::allocate(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:954:0
#12 0x0084b491 std::_Allocate_at_least_helper<class std::allocator<struct llvm::gi::MatchTableRecord>>(class std::allocator<struct llvm::gi::MatchTableRecord> &, unsigned int &) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:2184:0
#13 0x0087acf3 std::vector<struct llvm::gi::MatchTableRecord, class std::allocator<struct llvm::gi::MatchTableRecord>>::_Buy_raw(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:1992:0
#14 0x0087abcd std::vector<struct llvm::gi::MatchTableRecord, class std::allocator<struct llvm::gi::MatchTableRecord>>::_Buy_nonzero(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:2013:0
#15 0x0084d511 std::vector<struct llvm::gi::MatchTableRecord, class std::allocator<struct llvm::gi::MatchTableRecord>>::_Construct_n<struct llvm::gi::MatchTableRecord *const &, struct llvm::gi::MatchTableRecord *const &>(unsigned int, struct llvm::gi::MatchTableRecord *const &, struct llvm::gi::MatchTableRecord *const &) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:2070:0
#16 0x0086a89d std::vector<struct llvm::gi::MatchTableRecord, class std::allocator<struct llvm::gi::MatchTableRecord>>::vector<struct llvm::gi::MatchTableRecord, class std::allocator<struct llvm::gi::MatchTableRecord>>(class std::vector<struct llvm::gi::MatchTableRecord, class std::allocator<struct llvm::gi::MatchTableRecord>> const &) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:680:0
#17 0x0086b700 llvm::gi::MatchTable::MatchTable(class llvm::gi::MatchTable const &) (D:\Programmieren\C++\clang_build\llvm-project\build_ninja\bin\llvm-tblgen.exe+0xcb700)
#18 0x0082dd2e llvm::gi::MatchTable::buildTable(class llvm::ArrayRef<class llvm::gi::Matcher *>, bool, bool) D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\Common\GlobalISel\GlobalISelMatchTable.cpp:337:0
#19 0x00c838d4 `anonymous namespace'::GlobalISelEmitter::buildMatchTable D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\GlobalISelEmitter.cpp:2269:0
#20 0x00c7b74e `anonymous namespace'::GlobalISelEmitter::run D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\GlobalISelEmitter.cpp:2479:0
#21 0x00c85e7a llvm::TableGen::Emitter::OptClass<`anonymous namespace'::GlobalISelEmitter>::run D:\Programmieren\C++\clang_build\llvm-project\llvm\include\llvm\TableGen\TableGenBackend.h:40:0
#22 0x00a63def llvm::function_ref<(class llvm::RecordKeeper const &, class llvm::raw_ostream &)>::callback_fn<(class llvm::RecordKeeper const &, class llvm::raw_ostream &)>(int, class llvm::RecordKeeper const &, class llvm::raw_ostream &) D:\Programmieren\C++\clang_build\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:46:0
#23 0x00fb31c4 llvm::function_ref<(class llvm::RecordKeeper const &, class llvm::raw_ostream &)>::operator()(class llvm::RecordKeeper const &, class llvm::raw_ostream &) const D:\Programmieren\C++\clang_build\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:69:0
#24 0x00fb1379 llvm::TableGen::Emitter::ApplyCallback(class llvm::RecordKeeper const &, class llvm::raw_ostream &) D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\TableGen\TableGenBackend.cpp:69:0
#25 0x00fbb5dc llvm::TableGenMain(char const *, class std::function<(class llvm::raw_ostream &, class llvm::RecordKeeper const &)>) D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\TableGen\Main.cpp:138:0
#26 0x00dc1d1d main D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\TableGen.cpp:81:0
#27 0x01021f53 invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78:0
#28 0x01021dca _scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288:0
#29 0x01021c5d _scrt_common_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:331:0
#30 0x01021fd8 mainCRTStartup D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:17:0
#31 0x767dfcc9 (C:\Windows\System32\KERNEL32.DLL+0x1fcc9)
#32 0x7730809e (C:\Windows\SYSTEM32\ntdll.dll+0x6809e)
#33 0x7730806e (C:\Windows\SYSTEM32\ntdll.dll+0x6806e)
[1933/4145] Building AMDGPUGenRegisterInfo.inc...
ninja: build stopped: subcommand failed.```

[llvmbot]

@llvm/issue-subscribers-backend-amdgpu

Author: None (LukasReil)

I cloned llvm (ddef380) with git and followed the build process, as detailed in [the clang setup guide](https://clang.llvm.org/get_started.html). I used ninja, as detailed in `Using Ninja alongside Visual Studio` to build everything and then encountered the following crash:

[1916/4145] Building AMDGPUGenGlobalISel.inc...
FAILED: lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc D:/Programmieren/C++/clang_build/llvm-project/build_ninja/lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc
C:\Windows\system32\cmd.exe /C "cd /D D:\Programmieren\C++\clang_build\llvm-project\build_ninja && D:\Programmieren\C++\clang_build\llvm-project\build_ninja\bin\llvm-tblgen.exe -gen-global-isel -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU -ID:/Programmieren/C++/clang_build/llvm-project/build_ninja/include -ID:/Programmieren/C++/clang_build/llvm-project/llvm/include -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target --long-string-literals=0 D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU/AMDGPUGISel.td --write-if-changed -o lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc -d lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc.d"
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: D:\\Programmieren\\C++\\clang_build\\llvm-project\\build_ninja\\bin\\llvm-tblgen.exe -gen-global-isel -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU -ID:/Programmieren/C++/clang_build/llvm-project/build_ninja/include -ID:/Programmieren/C++/clang_build/llvm-project/llvm/include -I D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target --long-string-literals=0 D:/Programmieren/C++/clang_build/llvm-project/llvm/lib/Target/AMDGPU/AMDGPUGISel.td --write-if-changed -o lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc -d lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc.d
Exception Code: 0x80000003
 #<!-- -->0 0x00e861d9 HandleAbort D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\Support\Windows\Signals.inc:429:0
 #<!-- -->1 0x64342434 (C:\Windows\SYSTEM32\ucrtbased.dll+0xc2434)
 #<!-- -->2 0x64343a52 (C:\Windows\SYSTEM32\ucrtbased.dll+0xc3a52)
 #<!-- -->3 0x00e01871 llvm::report_bad_alloc_error(char const *, bool) D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\Support\ErrorHandling.cpp:177:0
 #<!-- -->4 0x00e0221f out_of_memory_new_handler D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\Support\ErrorHandling.cpp:190:0
 #<!-- -->5 0x64456388 (C:\Windows\SYSTEM32\MSVCP140D.dll+0x26388)
 #<!-- -->6 0x6432f962 (C:\Windows\SYSTEM32\ucrtbased.dll+0xaf962)
 #<!-- -->7 0x010204f7 operator new(unsigned int) D:\a\_work\1\s\src\vctools\crt\vcstartup\src\heap\new_scalar.cpp:41:0
 #<!-- -->8 0x008183fc std::_Default_allocate_traits::_Allocate(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:101:0
 #<!-- -->9 0x00809b1c std::_Allocate_manually_vector_aligned&lt;struct std::_Default_allocate_traits&gt;(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:158:0
#<!-- -->10 0x008099bf std::_Allocate&lt;8, struct std::_Default_allocate_traits&gt;(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:222:0
#<!-- -->11 0x008831e0 std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;::allocate(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:954:0
#<!-- -->12 0x0084b491 std::_Allocate_at_least_helper&lt;class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;&gt;(class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt; &amp;, unsigned int &amp;) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\xmemory:2184:0
#<!-- -->13 0x0087acf3 std::vector&lt;struct llvm::gi::MatchTableRecord, class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;&gt;::_Buy_raw(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:1992:0
#<!-- -->14 0x0087abcd std::vector&lt;struct llvm::gi::MatchTableRecord, class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;&gt;::_Buy_nonzero(unsigned int) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:2013:0
#<!-- -->15 0x0084d511 std::vector&lt;struct llvm::gi::MatchTableRecord, class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;&gt;::_Construct_n&lt;struct llvm::gi::MatchTableRecord *const &amp;, struct llvm::gi::MatchTableRecord *const &amp;&gt;(unsigned int, struct llvm::gi::MatchTableRecord *const &amp;, struct llvm::gi::MatchTableRecord *const &amp;) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:2070:0
#<!-- -->16 0x0086a89d std::vector&lt;struct llvm::gi::MatchTableRecord, class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;&gt;::vector&lt;struct llvm::gi::MatchTableRecord, class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;&gt;(class std::vector&lt;struct llvm::gi::MatchTableRecord, class std::allocator&lt;struct llvm::gi::MatchTableRecord&gt;&gt; const &amp;) C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.41.34120\include\vector:680:0
#<!-- -->17 0x0086b700 llvm::gi::MatchTable::MatchTable(class llvm::gi::MatchTable const &amp;) (D:\Programmieren\C++\clang_build\llvm-project\build_ninja\bin\llvm-tblgen.exe+0xcb700)
#<!-- -->18 0x0082dd2e llvm::gi::MatchTable::buildTable(class llvm::ArrayRef&lt;class llvm::gi::Matcher *&gt;, bool, bool) D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\Common\GlobalISel\GlobalISelMatchTable.cpp:337:0
#<!-- -->19 0x00c838d4 `anonymous namespace'::GlobalISelEmitter::buildMatchTable D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\GlobalISelEmitter.cpp:2269:0
#<!-- -->20 0x00c7b74e `anonymous namespace'::GlobalISelEmitter::run D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\GlobalISelEmitter.cpp:2479:0
#<!-- -->21 0x00c85e7a llvm::TableGen::Emitter::OptClass&lt;`anonymous namespace'::GlobalISelEmitter&gt;::run D:\Programmieren\C++\clang_build\llvm-project\llvm\include\llvm\TableGen\TableGenBackend.h:40:0
#<!-- -->22 0x00a63def llvm::function_ref&lt;(class llvm::RecordKeeper const &amp;, class llvm::raw_ostream &amp;)&gt;::callback_fn&lt;(class llvm::RecordKeeper const &amp;, class llvm::raw_ostream &amp;)&gt;(int, class llvm::RecordKeeper const &amp;, class llvm::raw_ostream &amp;) D:\Programmieren\C++\clang_build\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:46:0
#<!-- -->23 0x00fb31c4 llvm::function_ref&lt;(class llvm::RecordKeeper const &amp;, class llvm::raw_ostream &amp;)&gt;::operator()(class llvm::RecordKeeper const &amp;, class llvm::raw_ostream &amp;) const D:\Programmieren\C++\clang_build\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:69:0
#<!-- -->24 0x00fb1379 llvm::TableGen::Emitter::ApplyCallback(class llvm::RecordKeeper const &amp;, class llvm::raw_ostream &amp;) D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\TableGen\TableGenBackend.cpp:69:0
#<!-- -->25 0x00fbb5dc llvm::TableGenMain(char const *, class std::function&lt;(class llvm::raw_ostream &amp;, class llvm::RecordKeeper const &amp;)&gt;) D:\Programmieren\C++\clang_build\llvm-project\llvm\lib\TableGen\Main.cpp:138:0
#<!-- -->26 0x00dc1d1d main D:\Programmieren\C++\clang_build\llvm-project\llvm\utils\TableGen\TableGen.cpp:81:0
#<!-- -->27 0x01021f53 invoke_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78:0
#<!-- -->28 0x01021dca _scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288:0
#<!-- -->29 0x01021c5d _scrt_common_main D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:331:0
#<!-- -->30 0x01021fd8 mainCRTStartup D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:17:0
#<!-- -->31 0x767dfcc9 (C:\Windows\System32\KERNEL32.DLL+0x1fcc9)
#<!-- -->32 0x7730809e (C:\Windows\SYSTEM32\ntdll.dll+0x6809e)
#<!-- -->33 0x7730806e (C:\Windows\SYSTEM32\ntdll.dll+0x6806e)
[1933/4145] Building AMDGPUGenRegisterInfo.inc...
ninja: build stopped: subcommand failed.```
</details>

[arsenm]

Is there a more specific reason this is a bug, or just building on a very memory limited system? Is there some identified excessive usage somewhere?

[LukasReil]

It was built with a Ryzen 7 3700X with 32 GB RAM. According to Windows' Task Manager the entire System was asured somewhere around 80-90 GB, which was almost entirely used. There was almost nothing else open at that time, except for Firefox and some other programs, but nothing using up that much RAM, apart from the build process. I later built again, but with MSVC, which completed the build process without problems, although I think it also used a lot of RAM, but I built the BUILD_ALL configuration, so this was to be expected.

[jayfoad]

You probably need to set the cmake variable LLVM_PARALLEL_TABLEGEN_JOBS to limit the number to tablegen jobs that can run simultaneously.

Agreed that this is not really a bug. Sometimes running out of memory is inevitable.

[niziak]

You probably need to set the cmake variable LLVM_PARALLEL_TABLEGEN_JOBS to limit the number to tablegen jobs that can run simultaneously.

Agreed that this is not really a bug. Sometimes running out of memory is inevitable.

@jayfoad thx for hint with LLVM_PARALLEL_TABLEGEN_JOBS.

For others looking for exhausted RAM solution:

• LLVM_PARALLEL_TABLEGEN_JOBS is a new variable added in [CMake] Add tablegen job pool support #84762
• for Yocto it can be used like that: EXTRA_OECMAKE:append:pn-clang-native = " -DLLVM_PARALLEL_TABLEGEN_JOBS=${@oe.utils.cpu_count() // 2}"
• additionally you can limit architecture to built with -DLLVM_TARGETS_TO_BUILD="X86;AArch64"

Not LLVM bug.