Python 3.14/3.15a build aborting due to OOM during `test_functools` / `test_json`

[Thyre]

Bug report

Bug description:

In the context of EasyBuild, I'm trying to build Python 3.14.2 for a new toolchain based on GCC 15.2.0 (see this PR easybuilders/easybuild-easyconfigs#25006). Building Python worked fine on a few machines I've tested, all x86-64 based, with various distributions. However, builds consistently failed on two particular systems with similar hardware.

Looking into build failures, I was getting the following output:

0:00:20 load avg: 1.00 [18/43] test_functools
make: *** [Makefile:1020: profile-run-stamp] Killed

Checking the build command, test_functools is hanging indefinitely until seemingly killed by OOM. Checking closer with GDB, I'm getting this stack trace.

#79 0x000040000161fa1c in save_reduce (st=st@entry=0x40000153db10, self=self@entry=0x40002cb0c720, args=<optimized out>, obj=obj@entry=0x40002c6fbfa0) at ./Modules/_pickle.c:4273
#80 0x0000400001619154 in save (st=0x40000153db10, self=0x40002cb0c720, obj=0x40002c6fbfa0, pers_save=<optimized out>) at ./Modules/_pickle.c:4555
#81 0x0000400001616898 in store_tuple_elements (state=0x40000153db10, self=0x40002cb0c720, t=0x4000732a9240, len=1) at ./Modules/_pickle.c:2792
#82 0x000040000161a914 in save_tuple (state=0x40000153db10, self=0x40002cb0c720, obj=0x4000732a9240) at ./Modules/_pickle.c:2872
#83 save (st=st@entry=0x40000153db10, self=self@entry=0x40002cb0c720, obj=0x4000732a9240, pers_save=pers_save@entry=0) at ./Modules/_pickle.c:4434
#84 0x000040000161fa1c in save_reduce (st=st@entry=0x40000153db10, self=self@entry=0x40002cb0c720, args=<optimized out>, obj=obj@entry=0x40002c6fbfa0) at ./Modules/_pickle.c:4273
#85 0x0000400001619154 in save (st=0x40000153db10, self=0x40002cb0c720, obj=0x40002c6fbfa0, pers_save=<optimized out>) at ./Modules/_pickle.c:4555
#86 0x0000400001616898 in store_tuple_elements (state=0x40000153db10, self=0x40002cb0c720, t=0x4000732a9200, len=1) at ./Modules/_pickle.c:2792
#87 0x000040000161a914 in save_tuple (state=0x40000153db10, self=0x40002cb0c720, obj=0x4000732a9200) at ./Modules/_pickle.c:2872
#88 save (st=st@entry=0x40000153db10, self=self@entry=0x40002cb0c720, obj=0x4000732a9200, pers_save=pers_save@entry=0) at ./Modules/_pickle.c:4434

Checking with dmesg, the process was indeed killed with OOM. Having more than 400 GB of system memory, I would assume that this is sufficient to build Python

[18354.456973] [3169487]  9049 3169487     1340      616   458752        0             0 python3
[18354.465830] [3192788]  9049 3192788      216        0   458752        0             0 make
[18354.474416] [3206584]  9049 3206584  8964670  8952086 72220672        0             0 python
[18354.483181] [3207839]     0 3207839      145        0   393216        0             0 mmccrmonitor
[18354.492479] [3207840]     0 3207840      579      198   393216        0             0 psid
[18354.501065] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=nvidia-dcgm.service,mems_allowed=0-1,global_oom,task_memcg=/slurm/user-reuter1/job-14341053/step-0/tasks,task=python,pid=3206584,uid=9049
[18354.520291] Out of memory: Killed process 3206584 (python) total-vm:573738880kB, anon-rss:572924288kB, file-rss:0kB, shmem-rss:9216kB, UID:9049 pgtables:70528kB oom_score_adj:0

I've tried reducing the number at test_recursive_pickle yielding a, probably expected, test failure.

cpython/Lib/test/test_functools.py

Line 449 in df79316

with support.infinite_recursion(100):

The build then fails later on in test_json for the same reason.

#18 0x0000400001dbd5ec in encoder_listencode_obj (s=s@entry=0x40002cbabdc0, writer=writer@entry=0x40002c4422f0, obj=0x40002c4f47d0, indent_level=indent_level@entry=0, indent_cache=indent_cache@entry=0x0) at ./Modules/_json.c:1549
#19 0x0000400001dbda68 in encoder_listencode_list (s=0x40002cbabdc0, writer=0x40002c4422f0, seq=0x40013ab14680, indent_level=0, indent_cache=0x0) at ./Modules/_json.c:1805
#20 encoder_listencode_obj (s=s@entry=0x40002cbabdc0, writer=writer@entry=0x40002c4422f0, obj=obj@entry=0x40013ab14680, indent_level=indent_level@entry=0, indent_cache=indent_cache@entry=0x0) at ./Modules/_json.c:1519
#21 0x0000400001dbd70c in encoder_listencode_obj (s=s@entry=0x40002cbabdc0, writer=writer@entry=0x40002c4422f0, obj=0x40002c4f47d0, indent_level=indent_level@entry=0, indent_cache=indent_cache@entry=0x0) at ./Modules/_json.c:1560
#22 0x0000400001dbda68 in encoder_listencode_list (s=0x40002cbabdc0, writer=0x40002c4422f0, seq=0x40013ab14640, indent_level=0, indent_cache=0x0) at ./Modules/_json.c:1805
#23 encoder_listencode_obj (s=s@entry=0x40002cbabdc0, writer=writer@entry=0x40002c4422f0, obj=obj@entry=0x40013ab14640, indent_level=indent_level@entry=0, indent_cache=indent_cache@entry=0x0) at ./Modules/_json.c:1519

It's worth noting that builds on other platforms (Arch Linux, Fedora, Ubuntu 24) with x86 all worked out fine.

For testing, I've then tried to use GCC 14.3.0, yielding the same results.
Trying Python 3.13.5, the build passed. Python 3.14.1, 3.14.2 and 3.15.0a3 failed with the issues mentioned above.
Dependencies of Python were different between the two GCC versions, but still yielded the same result.

For builds, the following flags were used:

./configure --prefix=/tmp/software/Python/3.14.2-GCCcore-14.3.0  --build=aarch64-unknown-linux-gnu  --host=aarch64-unknown-linux-gnu  --enable-shared  --with-lto  --enable-optimizations  --with-ensurepip=upgrade

I've also tried using an external expat, with no noticeable differences.

---

Hardware information:

• Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, NVIDIA driver 580.95.05, Python 3.9.21
• Linux RHEL 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 570.133.20, Python 3.9.21

---

Unfortunately, I'm a bit stuck here. Looking through existing issues, I've found #113655, though this issue is related to Stack Overflows and Windows. I've also found PRs like #124264, but Python 3.14 removed these values altogether (#133080). I'm not sure if the issues I'm seeing are related.

Happy to provide more information, if needed.

CPython versions tested on:

3.15, 3.14, 3.13

Operating systems tested on:

Linux

Linked PRs

• gh-143460: Skip infinite recusion tests for infinite stack size #143606
• [3.14] gh-143460: Skip infinite recusion tests for infinite stack size (GH-143606) #143619

[picnixz]

I think the failure is because of PGO configuration:

0:00:20 load avg: 1.00 [18/43] test_functools

We are executing a limited subset of tests to gather profiling data and this could be the reason. I think your tool chain may be having issues with that. Have you tried testing this without enabling optimizations? (that is, try without --with-lto and without --enable-optimizations and then only try --enable-optimizations but without LTO)

[Fidget-Spinner]

@picnixz no I don't think it's the PGO's fault. The problem is more likely that our stack detection algorithms are not working on AArch64 Rocky Linux. The fact the OP has infinite recursion in some of the tests that don't terminate is a sign that the Python-to-Python calls are heap allocating more frames and skipping both the C stack and recursion protectors.

@Thyre please try checking which ifdef compiles on your system. If it's the last one, that probably means it's broken, as it should be using __builtin_frame_address for GNU/Linux.
https://github.com/python/cpython/blob/main/Include/internal/pycore_pystate.h#L322

You can test for example by adding Py_FatalError("no stack protection") on the last ifdef's else clause, or a printf also works.

[Thyre]

Thanks for the quick follow up 😄

I've tried adding Py_FatalError("no stack protection") to the last ifdef's else clause, but this didn't trigger. Next, I checked with this modification:

static inline uintptr_t
_Py_get_machine_stack_pointer(void) {
#if _Py__has_builtin(__builtin_frame_address) || defined(__GNUC__)
    Py_FatalError("uses __builtin_frame_address");
    return (uintptr_t)__builtin_frame_address(0);
#elif defined(_MSC_VER)
    return (uintptr_t)_AddressOfReturnAddress();
#else
    char here;
    /* Avoid compiler warning about returning stack address */
    return return_pointer_as_int(&here);
#endif
}

which then failed with the debug message:

lto-wrapper: warning: using serial compilation of 8 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
./Programs/_freeze_module getpath ./Modules/getpath.py Python/frozen_modules/getpath.h
./Programs/_freeze_module importlib._bootstrap ./Lib/importlib/_bootstrap.py Python/frozen_modules/importlib._bootstrap.h
./Programs/_freeze_module importlib._bootstrap_external ./Lib/importlib/_bootstrap_external.py Python/frozen_modules/importlib._bootstrap_external.h
./Programs/_freeze_module zipimport ./Lib/zipimport.py Python/frozen_modules/zipimport.h
Fatal Python error: _Py_get_machine_stack_pointer: uses __builtin_frame_address
Python runtime state: preinitialized

Current thread 0x0000400029e567e0 [_freeze_module] (most recent call first):
  <no Python frame>
make[2]: *** [Makefile:1928: Python/frozen_modules/importlib._bootstrap.h] Aborted (core dumped)
make[2]: *** Waiting for unfinished jobs....
Fatal Python error: _Py_get_machine_stack_pointer: uses __builtin_frame_address
Python runtime state: preinitialized

Current thread 0x00004000334b67e0 [_freeze_module] (most recent call first):
  <no Python frame>
Fatal Python error: _Py_get_machine_stack_pointer: uses __builtin_frame_address
Python runtime state: preinitialized

Current thread 0x00004000398267e0 [_freeze_module] (most recent call first):
  <no Python frame>
Fatal Python error: _Py_get_machine_stack_pointer: uses __builtin_frame_address
Python runtime state: preinitialized

Current thread 0x0000400034d967e0 [_freeze_module] (most recent call first):
  <no Python frame>
make[2]: *** [Makefile:1923: Python/frozen_modules/getpath.h] Aborted (core dumped)
make[2]: *** [Makefile:1931: Python/frozen_modules/importlib._bootstrap_external.h] Aborted (core dumped)
make[2]: *** [Makefile:1934: Python/frozen_modules/zipimport.h] Aborted (core dumped)
make[2]: Leaving directory '/dev/shm/reuter1/easybuild/build/Python/3.14.2/GCCcore-15.2.0/Python-3.14.2'

So __builtin_frame_address should be used.

---

To make sure this isn't caused by EasyBuild's set flags in some way, I also tried a build just with the systems GCC, i.e. gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5). That ended in the exact same result.

A test build without --enable-optimizations worked. That's not surprising though, as the tests are not executed then. Executing the same test case afterwards causes the same failure. So PGO is not involved here. Removing --with-lto has no impact as well.

A verbose run unfortunately doesn't give too much additional information:

[reuter1@jrc0900 Python-3.14.2]$ LD_LIBRARY_PATH=$(pwd):LD_LIBRARY_PATH ./python Lib/test/test_functools.py --verbose
[...]
test_pickle (__main__.TestPartialC.test_pickle) ... ok
test_placeholders (__main__.TestPartialC.test_placeholders) ... ok
test_placeholders_kw_restriction (__main__.TestPartialC.test_placeholders_kw_restriction) ... ok
test_placeholders_optimization (__main__.TestPartialC.test_placeholders_optimization) ... ok
test_placeholders_refcount_smoke (__main__.TestPartialC.test_placeholders_refcount_smoke) ... ok
test_placeholders_trailing_raise (__main__.TestPartialC.test_placeholders_trailing_raise) ... ok
test_positional (__main__.TestPartialC.test_positional) ... ok
test_protection_of_callers_dict_argument (__main__.TestPartialC.test_protection_of_callers_dict_argument) ... ok
test_recursive_pickle (__main__.TestPartialC.test_recursive_pickle) ... 
Program terminated with signal SIGKILL, Killed.
The program no longer exists.

[Thyre]

I think this might be related to the ulimit stack size. On both systems, the stack size is set to unlimited by default. Limiting it to e.g. 8192 via ulimit -s 8192, the tests pass almost immediately.

I need to check this on the other systems, once I have access to them again.

---

A test build with ulimit -s 16384 worked just fine with EasyBuild as well (see easybuilders/easybuild-easyconfigs#25006 (comment)).

---

A second test build on x86 (Ubuntu 24.04, Intel Core i7-1260P, 32GB RAM) successfully (?) aborted with ulimit -s unlimited set. So I'm pretty sure this is the culprit.

LD_PRELOAD=/opt/EasyBuild/apps/build/Python/3.14.2/GCCcore-15.2.0/Python-3.14.2/libpython3.14.so ./python -m test --pgo --timeout=
Using random seed: 3222210463
0:00:00 load avg: 1.32 Run 43 tests sequentially in a single process
0:00:00 load avg: 1.32 [ 1/43] test_array
0:00:01 load avg: 1.32 [ 1/43] test_array passed
0:00:01 load avg: 1.32 [ 2/43] test_base64
0:00:01 load avg: 1.30 [ 2/43] test_base64 passed
[...]
0:00:20 load avg: 1.16 [18/43] test_functools
Killed
make: *** [Makefile:1016: profile-run-stamp] Error 137

[Fidget-Spinner]

I'm not sure if Python should support ulimit -s unlimited. So I'm closing this issue for now. Thanks for finding out the root cause!

[Krzmbrzl]

My goodness - thanks for the hint with ulimit! I was letting the test run forever and it just wouldn't finish. Turns out I was simply running it on a machine with lots of memory and wasn't patient enough to let it run out of it to see the error reported here.

@Fidget-Spinner While I don't know whether ulimit -s unlimited is worth supporting or not, I think it would be very valuable to check for this condition and at least output a warning (or even straight up an error) when detected. Otherwise, people will likely have a hard time when trying to figure out why it's not working 🤔