[Python] pyarrow 21.0.0 binary mismatch with raspberry 4 hardware

[epigramx]

Describe the bug, including details regarding any error messages, version, and platform.

Details are here: freqtrade/freqtrade#12047

Basically the default binary core dumbs on that hardware because of an illegal instruction.

PS Forgive me if it's the wrong repository for this report but this is where the pyarrow page directs

Component(s)

Python

[amoeba]

Hi @epigramx reporting here is fine. I can reproduce on an aarch64 pi,

pi@pi:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
pi@pi:~$ arch
aarch64
pi@pi:~$ python3 -m pip install pyarrow
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pyarrow
  Downloading pyarrow-21.0.0-cp39-cp39-manylinux_2_28_aarch64.whl (41.1 MB)
     |████████████████████████████████| 41.1 MB 4.2 kB/s
Installing collected packages: pyarrow
Successfully installed pyarrow-21.0.0
pi@pi:~$ python
Python 3.9.2 (default, Mar 20 2025, 02:07:39)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
Illegal instruction

Full backtrace

(gdb) thread apply all bt

Thread 1 (Thread 0x7ff7cee040 (LWP 338998) "python"):
#0  0x0000007ff4aa947c in _mi_options_init () from /home/pi/.local/lib/python3.9/site-packages/pyarrow/libarrow.so.2100
#1  0x0000007ff4aa7818 in _mi_auto_process_init () from /home/pi/.local/lib/python3.9/site-packages/pyarrow/libarrow.so.2100
#2  0x0000007ff7fdac9c in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7ffffffb38, env=env@entry=0x7ffffffb48) at dl-init.c:72
#3  0x0000007ff7fdada4 in call_init (env=0x7ffffffb48, argv=0x7ffffffb38, argc=1, l=<optimized out>) at dl-init.c:30
#4  _dl_init (main_map=0xa3adb0, argc=1, argv=0x7ffffffb38, env=0x7ffffffb48) at dl-init.c:119
#5  0x0000007ff7dfb1c4 in __GI__dl_catch_exception (exception=0x0, operate=0x7ff7fde130 <call_dl_init>, args=0x7fffffca50) at dl-error-skeleton.c:182
#6  0x0000007ff7fded28 in dl_open_worker (a=a@entry=0x7fffffcca0) at dl-open.c:758
#7  0x0000007ff7dfb164 in __GI__dl_catch_exception (exception=0x7fffffcc88, operate=0x7ff7fde9b0 <dl_open_worker>, args=0x7fffffcca0) at dl-error-skeleton.c:208
#8  0x0000007ff7fde610 in _dl_open (file=0x7ff76f75d0 "/home/pi/.local/lib/python3.9/site-packages/pyarrow/lib.cpython-39-aarch64-linux-gnu.so", mode=-2147483646, caller_dlopen=0x603740, nsid=-2, argc=1, argv=0x7ffffffb38, env=0x7ffffffb48) at dl-open.c:837
#9  0x0000007ff7f880b4 in dlopen_doit (a=a@entry=0x7fffffcf58) at dlopen.c:66
#10 0x0000007ff7dfb164 in __GI__dl_catch_exception (exception=exception@entry=0x7fffffcee0, operate=0x7ff7f88050 <dlopen_doit>, args=0x7fffffcf58) at dl-error-skeleton.c:208
#11 0x0000007ff7dfb230 in __GI__dl_catch_error (objname=0x97c5a0, errstring=0x97c5a8, mallocedp=0x97c598, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:227
#12 0x0000007ff7f88960 in _dlerror_run (operate=operate@entry=0x7ff7f88050 <dlopen_doit>, args=args@entry=0x7fffffcf58) at dlerror.c:170
#13 0x0000007ff7f88154 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#14 0x0000000000603740 in ?? ()
#15 0x0000000000554ed8 in ?? ()
#16 0x00000000004b4cbc in ?? ()
#17 0x000000000049b998 in _PyEval_EvalFrameDefault ()
#18 0x00000000004985d8 in ?? ()
#19 0x00000000004b41c0 in _PyFunction_Vectorcall ()
#20 0x000000000049e3a8 in _PyEval_EvalFrameDefault ()
#21 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#22 0x000000000049a318 in _PyEval_EvalFrameDefault ()
#23 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#24 0x000000000049a164 in _PyEval_EvalFrameDefault ()
#25 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#26 0x000000000049a164 in _PyEval_EvalFrameDefault ()
#27 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#28 0x000000000049a164 in _PyEval_EvalFrameDefault ()
#29 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#30 0x00000000004b32f0 in ?? ()
#31 0x00000000004cdfd0 in _PyObject_CallMethodIdObjArgs ()
#32 0x00000000004cccf4 in PyImport_ImportModuleLevelObject ()
#33 0x000000000049c0a0 in _PyEval_EvalFrameDefault ()
#34 0x00000000004985d8 in ?? ()
#35 0x0000000000498374 in _PyEval_EvalCodeWithName ()
#36 0x0000000000498308 in PyEval_EvalCode ()
#37 0x00000000004d7dec in ?? ()
#38 0x00000000004b4cbc in ?? ()
#39 0x000000000049b998 in _PyEval_EvalFrameDefault ()
#40 0x00000000004985d8 in ?? ()
#41 0x00000000004b41c0 in _PyFunction_Vectorcall ()
#42 0x000000000049e3a8 in _PyEval_EvalFrameDefault ()
#43 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#44 0x000000000049a318 in _PyEval_EvalFrameDefault ()
#45 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#46 0x000000000049a164 in _PyEval_EvalFrameDefault ()
#47 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#48 0x000000000049a164 in _PyEval_EvalFrameDefault ()
#49 0x00000000004b3f68 in _PyFunction_Vectorcall ()
#50 0x00000000004b32f0 in ?? ()
#51 0x00000000004cdfd0 in _PyObject_CallMethodIdObjArgs ()
#52 0x00000000004cccf4 in PyImport_ImportModuleLevelObject ()
#53 0x000000000049c0a0 in _PyEval_EvalFrameDefault ()
#54 0x00000000004985d8 in ?? ()
#55 0x0000000000498374 in _PyEval_EvalCodeWithName ()
#56 0x0000000000498308 in PyEval_EvalCode ()
#57 0x000000000060252c in ?? ()
#58 0x00000000005fe6a0 in ?? ()
#59 0x0000000000421130 in ?? ()
#60 0x0000000000420dd0 in PyRun_InteractiveLoopFlags ()
#61 0x0000000000601be0 in PyRun_AnyFileExFlags ()
#62 0x00000000005f69fc in Py_RunMain ()
#63 0x00000000005d60e8 in Py_BytesMain ()
#64 0x0000007ff7d10dd8 in __libc_start_main (main=0x5d5fe8 <_start+56>, argc=1, argv=0x7ffffffb38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#65 0x00000000005d5fe4 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[gobes]

Hi, the illegal instruction in pyarrow 21.0.0 also affect two march=armv8-a variants

Contex (freqtrade 2025.7, PYTHON_VERSION="3.13")

Raspberry Pi 4 Model B Rev 1.4

• CPU armv8-a/cortex-a72
• Debian GNU/Linux 12 (bookworm)
• Linux raspberrypi 6.12.25+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.25-1+rpt1 (2025-04-30) aarch64 GNU/Linux

Nvidia Jetson Nano 4GB dev board

• CPU armv8-a/cortex-a57
• Ubuntu 20.04.6 LTS
• Linux jetson 4.9.337-tegra #1 SMP PREEMPT Mon Nov 4 23:41:41 PST 2024 aarch64 aarch64 aarch64 GNU/Linux

Any adjustment of -DARROW_SIMD_* define does not help.

Instead, rebuild arrow and pyarrow as bellow solved the illegal instruction on both platforms:

--build-arg CC="gcc -march=armv8-a -mtune=cortex-a57 -mno-outline-atomics
--build-arg CXX="g++ -march=armv8-a -mtune=cortex-a57 -mno-outline-atomics

--build-arg CC="gcc -march=armv8-a -mtune=cortex-a72 -mno-outline-atomics
--build-arg CXX="g++ -march=armv8-a -mtune=cortex-a72 -mno-outline-atomics

Pass extra flags to arrow/cpp/release
-DMI_USE_LIBATOMIC=ON
-DARROW_MIMALLOC=OFF

Pass extra flags to pyarrow
ENV MI_USE_LIBATOMIC=ON
ENV DARROW_MIMALLOC=OFF

It seems like the illegal instruction is produced by

(gdb) x/i $pc
=> 0x7f8d35f6a0 <mi_out_buf_flush+64>: ldaddal x1, x1, [x0]

Full gdb backtrace

gdb on main branch 

GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...
(No debugging symbols found in python3)
Starting program: /usr/local/bin/python3 -c import\ pyarrow
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0x0000007f836cec4c in _mi_options_init () from /usr/local/lib/python3.13/site-packages/pyarrow/libarrow.so.2100
(gdb) bt full 
#0  0x0000007f836cec4c in _mi_options_init () from /usr/local/lib/python3.13/site-packages/pyarrow/libarrow.so.2100
No symbol table info available.
#1  0x0000007f836ccfe8 in _mi_auto_process_init () from /usr/local/lib/python3.13/site-packages/pyarrow/libarrow.so.2100
No symbol table info available.
#2  0x0000007f84aaa448 in call_init (env=0x7fe664f798, argv=0x7fe664f778, argc=3, l=<optimized out>) at ./elf/dl-init.c:74
        j = 0
        jm = <optimized out>
        addrs = <optimized out>
        init_array = <optimized out>
        __PRETTY_FUNCTION__ = "call_init"
        init_array = <optimized out>
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
#3  call_init (l=<optimized out>, argc=3, argv=0x7fe664f778, env=0x7fe664f798) at ./elf/dl-init.c:26
        init_array = <optimized out>
        __PRETTY_FUNCTION__ = "call_init"
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
#4  0x0000007f84aaa554 in _dl_init (main_map=0x558e7c00f0, argc=3, argv=0x7fe664f778, env=0x7fe664f798) at ./elf/dl-init.c:121
        preinit_array = 0x0
        preinit_array_size = <optimized out>
        i = <optimized out>
#5  0x0000007f844aea44 in __GI__dl_catch_exception (exception=0x0, operate=0x7f84ab0180 <call_dl_init>, args=0x7fe664dc90) at ./elf/dl-error-skeleton.c:182
        old = <optimized out>
        errcode = 127
        c = {exception = 0x7fe664dc90, errcode = 0x7fe664db0c, env = {{__jmpbuf = {184, 547686815584, 547686866944, 549326216336, 547683136008, 3, 549326223224, 549326223256, 2147483650, 547683780856, 
                549326215872, 16624411701779815989, 547683754624, 16624411700199746837, 0, 0, 0, 0, 0, 0, 0, 0}, __mask_was_saved = 0, __saved_mask = {__val = {547682143840, 6053859657349367846, 549326216416, 
                  549326216416, 549326216384, 18446743528248704992, 4596253733475294464, 549326216432, 547682813888, 547675063168, 2, 549326216288, 547686646592, 549326216288, 547686646632, 549326216904}}}}}
        old = <optimized out>
#6  0x0000007f84ab03b8 in dl_open_worker (a=a@entry=0x7fe664dec8) at ./elf/dl-open.c:808
        init_args = {new = 0x558e7c00f0, argc = 3, argv = 0x7fe664f778, env = 0x7fe664f798}
        args = 0x7fe664dec8
        mode = -2147483646
        new = 0x558e7c00f0
#7  0x0000007f844ae9ec in __GI__dl_catch_exception (exception=0x7fe664deb0, operate=0x7f84ab0310 <dl_open_worker>, args=0x7fe664dec8) at ./elf/dl-error-skeleton.c:208
        errcode = 127
        c = {exception = 0x7fe664deb0, errcode = 0x7fe664dcfc, env = {{__jmpbuf = {184, 547686815584, 547686866944, 549326216880, 547683136008, 3, 549326223224, 549326223256, 2147483650, 547683780856, 
                549326216368, 16624411701779815989, 549326216936, 16624411700199746405, 0, 0, 0, 0, 0, 0, 0, 0}, __mask_was_saved = 0, __saved_mask = {__val = {0, 547685657856, 549326216808, 0, 547681866960, 
                  549326216872, 547674989472, 1, 547674989472, 4583450494, 547677061184, 549326216784, 547686647576, 18446744073709551614, 547674869200, 547686866944}}}}}
        old = 0x7fe664df80
#8  0x0000007f84ab075c in _dl_open (file=0x7f83f74dd0 "/usr/local/lib/python3.13/site-packages/pyarrow/lib.cpython-313-aarch64-linux-gnu.so", mode=-2147483646, caller_dlopen=0x7f84757208, nsid=-2, argc=3, 
    argv=0x7fe664f778, env=0x7fe664f798) at ./elf/dl-open.c:884
        args = {file = 0x7f83f74dd0 "/usr/local/lib/python3.13/site-packages/pyarrow/lib.cpython-313-aarch64-linux-gnu.so", mode = -2147483646, caller_dlopen = 0x7f84757208, map = 0x558e7c00f0, nsid = 0, 
          original_global_scope_pending_adds = 0, libc_already_loaded = true, worker_continue = true, argc = 3, argv = 0x7fe664f778, env = 0x7fe664f798}
        exception = {objname = 0x7f84ae1428 "\206\214T\204\177", errstring = 0x1 <error: Cannot access memory at address 0x1>, message_buffer = 0x7f84ae0210 ""}
        errcode = <optimized out>
        __PRETTY_FUNCTION__ = "_dl_open"
--Type <RET> for more, q to quit, c to continue without paging--
#9  0x0000007f843fae54 in dlopen_doit (a=a@entry=0x7fe664e1a8) at ./dlfcn/dlopen.c:56
        args = 0x7fe664e1a8
#10 0x0000007f844ae9ec in __GI__dl_catch_exception (exception=exception@entry=0x7fe664e100, operate=0x7f843fadf0 <dlopen_doit>, args=0x7fe664e1a8) at ./elf/dl-error-skeleton.c:208
        errcode = 127
        c = {exception = 0x7fe664e100, errcode = 0x7fe664df7c, env = {{__jmpbuf = {184, 547686815584, 549326217592, 549326217688, 547686865824, 547675040304, 547686815584, 547674989888, 549326217944, 
                547683780856, 549326217008, 16624411701779815989, 547686843840, 16624411700199745765, 0, 0, 0, 0, 0, 0, 0, 0}, __mask_was_saved = 0, __saved_mask = {__val = {547686844288, 547674921920, 
                  4596253733475294464, 549326217472, 547681839548, 549326218480, 547686672268, 0, 0, 547674869200, 549326217688, 547686117056, 547675040304, 549326217632, 549326217680, 549326217680}}}}}
        old = 0x0
#11 0x0000007f844aeab0 in __GI__dl_catch_error (objname=0x7fe664e178, errstring=0x7fe664e180, mallocedp=0x7fe664e177, operate=<optimized out>, args=<optimized out>) at ./elf/dl-error-skeleton.c:227
        exception = {objname = 0xfffffffd <error: Cannot access memory at address 0xfffffffd>, errstring = 0xc8 <error: Cannot access memory at address 0xc8>, 
          message_buffer = 0x7f843f89f0 <__GI__IO_default_xsputn> "\202\b"}
        errorcode = <optimized out>
#12 0x0000007f843fa8e0 in _dlerror_run (operate=operate@entry=0x7f843fadf0 <dlopen_doit>, args=args@entry=0x7fe664e1a8) at ./dlfcn/dlerror.c:138
        result = <optimized out>
        objname = 0xff0000 <error: Cannot access memory at address 0xff0000>
        errstring = 0xf00ffff0000 <error: Cannot access memory at address 0xf00ffff0000>
        malloced = 255
        errcode = <optimized out>
#13 0x0000007f843faf28 in dlopen_implementation (dl_caller=<optimized out>, mode=<optimized out>, file=<optimized out>) at ./dlfcn/dlopen.c:71
        args = {file = 0x7f83f74dd0 "/usr/local/lib/python3.13/site-packages/pyarrow/lib.cpython-313-aarch64-linux-gnu.so", mode = 2, new = 0xd81000d3100dd80c, caller = 0x7f84757208}
        args = <optimized out>
#14 ___dlopen (file=<optimized out>, mode=<optimized out>) at ./dlfcn/dlopen.c:81
No locals.
#15 0x0000007f84757208 in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#16 0x0000007f846671d0 in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#17 0x0000007f84648470 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#18 0x0000007f84663f9c in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#19 0x0000007f84691804 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#20 0x0000007f8469055c in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#21 0x0000007f8464a04c in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#22 0x0000007f846f6458 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#23 0x0000007f8471974c in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#24 0x0000007f8465ec78 in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#25 0x0000007f84648470 in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#26 0x0000007f84663f9c in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#27 0x0000007f84691804 in PyObject_CallMethodObjArgs () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#28 0x0000007f8469055c in PyImport_ImportModuleLevelObject () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#29 0x0000007f8464a04c in _PyEval_EvalFrameDefault () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#30 0x0000007f846f6458 in PyEval_EvalCode () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#31 0x0000007f84755cc4 in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#32 0x0000007f84751e54 in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#33 0x0000007f8474b9a8 in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#34 0x0000007f8474b810 in ?? () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#35 0x0000007f8474986c in Py_RunMain () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#36 0x0000007f846df910 in Py_BytesMain () from /usr/local/bin/../lib/libpython3.13.so.1.0
No symbol table info available.
#37 0x0000007f843a7740 in __libc_start_call_main (main=main@entry=0x556b4bc960, argc=argc@entry=3, argv=argv@entry=0x7fe664f778) at ../sysdeps/nptl/libc_start_call_main.h:58
        self = <optimized out>
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {549326223224, 3, 366872477104, 366872349024, 549326223256, 547686865824, 0, 547686866984, 366872477104, 0, 549326222848, 16624411701774532821, 
                7017564186559734879, 16624411700199735765, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7f84ae5ba0 <_rtld_global_ro>, 0x7f84520080 <_dl_audit_preinit@got.plt>}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = -2068948064}}}
        not_first_call = <optimized out>
#38 0x0000007f843a7818 in __libc_start_main_impl (main=0x556b4bc960, argc=3, argv=0x7fe664f778, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>)
    at ../csu/libc-start.c:360
No locals.
#39 0x000000556b4bc870 in _start ()
No symbol table info available.
(gdb) disassemble $pc-32,$pc+32
Dump of assembler code from 0x7f836cec2c to 0x7f836cec6c:
   0x0000007f836cec2c:	nop
   0x0000007f836cec30 <_mi_options_init+0>:	stp	x29, x30, [sp, #-48]!
   0x0000007f836cec34 <_mi_options_init+4>:	mov	x29, sp
   0x0000007f836cec38 <_mi_options_init+8>:	stp	x19, x20, [sp, #16]
   0x0000007f836cec3c <_mi_options_init+12>:	adrp	x20, 0x7f83901000 <_ZZN5arrow8internal13GetRandomSeedEvE8seed_gen+2208>
   0x0000007f836cec40 <_mi_options_init+16>:	add	x0, x20, #0x5f0
   0x0000007f836cec44 <_mi_options_init+20>:	stp	x21, x22, [sp, #32]
   0x0000007f836cec48 <_mi_options_init+24>:	mov	x22, #0x1                   	// #1
=> 0x0000007f836cec4c <_mi_options_init+28>:	ldaddal	x22, x22, [x0]
   0x0000007f836cec50 <_mi_options_init+32>:	cmp	x22, #0x4, lsl #12
   0x0000007f836cec54 <_mi_options_init+36>:	mov	x0, #0x4000                	// #16384
   0x0000007f836cec58 <_mi_options_init+40>:	adrp	x19, 0x7f83901000 <_ZZN5arrow8internal13GetRandomSeedEvE8seed_gen+2208>
   0x0000007f836cec5c <_mi_options_init+44>:	csel	x22, x22, x0, ls	// ls = plast
   0x0000007f836cec60 <_mi_options_init+48>:	add	x0, x19, #0x630
   0x0000007f836cec64 <_mi_options_init+52>:	strb	wzr, [x0, x22]
   0x0000007f836cec68 <_mi_options_init+56>:	ldrb	w1, [x19, #1584]
End of assembler dump.
(gdb) info registers
x0             0x7f839015f0        547668104688
x1             0x7f84ae5b98        547686865816
x2             0x0                 0
x3             0x0                 0
x4             0x16db81f9          383484409
x5             0x9873cb36          2557725494
x6             0x177f654a          394224970
x7             0xeae074fe          3940578558
x8             0x7f838fa678        547668076152
x9             0x9fd501d1          2681536977
x10            0x12bdd8f8          314431736
x11            0x197e461e          427705886
x12            0x2e0644f5          772162805
x13            0xa8263ded          2821078509
x14            0x13c87c2           20744130
x15            0xfdf4f1ed          4260688365
x16            0x7f838f2420        547668042784
x17            0x7f84401640        547679639104
x18            0x7fe664d9b0        549326215600
x19            0x7f838fa000        547668074496
x20            0x7f83901000        547668103168
x21            0x7fe664f778        549326223224
x22            0x1                 1
x23            0x7f838ba4b0        547667813552
x24            0x0                 0
x25            0x0                 0
x26            0x7fe664f798        549326223256
x27            0x80000002          2147483650
x28            0x7f847f48f8        547683780856
x29            0x7fe664d9d0        549326215632
x30            0x7f836ccfe8        547665793000
sp             0x7fe664d9d0        0x7fe664d9d0
pc             0x7f836cec4c        0x7f836cec4c <_mi_options_init+28>
cpsr           0x60000000          [ EL=0 BTYPE=0 C Z ]
fpsr           0x10                [ IXC ]
fpcr           0x0                 [ Len=0 Stride=0 RMode=0 ]
tpidr          0x7f84ad9760        0x7f84ad9760
tpidr2         0x0                 0x0
(gdb)  x/i $pc
=> 0x7f836cec4c <_mi_options_init+28>:	ldaddal	x22, x22, [x0]

armv8-a/cortex-a72 details

cat /proc/cpuinfo 
processor	: 0
BogoMIPS	: 108.00
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

. . .

Revision	: d03114
Serial		: 10000000775b6cfe
Model		: Raspberry Pi 4 Model B Rev 1.4




lscpu

Architecture:             aarch64
  CPU op-mode(s):         32-bit, 64-bit
  Byte Order:             Little Endian
CPU(s):                   4
  On-line CPU(s) list:    0-3
Vendor ID:                ARM
  Model name:             Cortex-A72
    Model:                3
    Thread(s) per core:   1
    Core(s) per cluster:  4
    Socket(s):            -
    Cluster(s):           1
    Stepping:             r0p3
    CPU(s) scaling MHz:   100%
    CPU max MHz:          1800.0000
    CPU min MHz:          600.0000
    BogoMIPS:             108.00
    Flags:                fp asimd evtstrm crc32 cpuid
Caches (sum of all):      
  L1d:                    128 KiB (4 instances)
  L1i:                    192 KiB (4 instances)
  L2:                     1 MiB (1 instance)

armv8-a/cortex-a57 details

lscpu 

Architecture:        aarch64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
Vendor ID:           ARM
Model:               1
Model name:          Cortex-A57
Stepping:            r1p1
CPU max MHz:         1479.0000
CPU min MHz:         102.0000
BogoMIPS:            38.40
L1d cache:           128 KiB
L1i cache:           192 KiB
L2 cache:            2 MiB
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32


cat /proc/cpuinfo

processor	: 0
model name	: ARMv8 Processor rev 1 (v8l)
BogoMIPS	: 38.40
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x1
CPU part	: 0xd07
CPU revision	: 1

...

It seems that both ARMv8.0 variants do not support the pyarrow whell prebuild arch64 version, and so, without knowing your perspective on supporting older CPU versions, I wonder if you intend to support them on https://pypi.org/project/pyarrow/ or it's up to us (consumers) to build it?

[xmatthias]

21.0.0 changelogs don't hint at this being an intentional change - so i think this is an (accidental) regression between 20.0.0 and 21.0.0.

it also doesn't seem to impact all aarch64 platforms - as i can use pyarrow 21.0.0 on arm64 oracle VPS's just fine.

[gobes]

It's seems that ldaddal instruction is supported in Supported in the Arm®v8.1 architecture and later. , but the above two cortex are Arm®v8.0

[amoeba]

Thanks for the above context, this is all very helpful.

These platforms aren't explicitly supported so users would be expected to build binaries themselves. I realize it was also not explicit that we do not support them and that we implicitly did support them for some time. Explicit support is essentially defined by what we test in CI before releases in and we plan to have a structured support policy in #46002. A policy for PyArrow would follow.

Two things that would help to know are:

1. Are there good free/cheap CI environments for these older Arm variants?
2. How are other Python projects handling supporting these platforms?

[epigramx]

expected to build binaries themselves

This causes big issues for those platforms because they are very low power so building anything takes about 20 times longer than a typical low cost PC (especially since it will often trigger another dependency to be built on top of it).

Aren't binaries nowadays able to support multiple instruction sets so the faster processors don't suffer in performance?

A core issue is that simpler devices aren't simpler because they're older but because it's helpful to be low power/cost.

[amoeba]

I agree that it's an issue for the use case. If we can find a way to make the aarch64 wheels more compatible that would be good. Increasing coverage in CI would be good too and sort of a pre-requisite. Finding out what other python packages do about this would be a good place to start.

Aren't binaries nowadays able to support multiple instruction sets so the faster processors don't suffer in performance?

I'm not sure.

[xmatthias]

i don't think you can generalize this for "binaries" per se (across languages) - but for wheels - i think the expectation of an aarch64 wheel is that it's compatible with all systems that identify as aarch64 (making the current pyarrow wheels an exception / deviation from this).

The problem with python wheels is - by having wheels available for a matching platform on pypi, it's actually pretty difficult (requires additional flags) to NOT install the wheel if you're on that platform - resulting in a broken build as pyarrow is not working if installed using pip install pyarrow (which i think is the default for python installs).

---

Now from a quick search (non-authoritive, non-exhaustive) - i've found uqfoundation/multiprocess#75 (comment) - which assumes that wheels should be build against armv8.0 for maximum compatibility.
pypi / pip doesn't allow to specify this further - as in "aarch64-arm8.0", "aarch64-arm8.1" - so i think running/compiling against the baseline is the best thing from a compatibility point of view (not sure on the impact of this for newer processors of not using this specific feature?)