[Python][C++] pyarrow._compute Contains AVX Instructions During Import

[Alvaro-Kothe]

Describe the bug, including details regarding any error messages, version, and platform.

According to the Arrow C++ environment variables documentation, the default SIMD level is SSE4.2. However, importing pyarrow._compute executes AVX instructions even when running on a CPU that only supports up to SSE4.2.

Reproduction

Setup venv and install pyarrow from PyPI:

$ python -m venv venv && venv/bin/pip install --no-cache pyarrow
... Successfully installed pyarrow-23.0.1

Then run the Python interpreter under QEMU emulating a Westmere CPU (which supports SSE4.2 but not AVX):

$ qemu-x86_64-static -cpu Westmere venv/bin/python -c 'import pyarrow._compute'
qemu: uncaught target signal 4 (Illegal instruction) - core dumped

The disassembly of the coredump shows that it uses AVX instructions.

(gdb) disassemble $pc-16,$pc+16
Dump of assembler code from 0x7f2ff65ceb32 to 0x7f2ff65ceb52:
   0x00007f2ff65ceb32:  push   r12
   0x00007f2ff65ceb34:  push   rbp
   0x00007f2ff65ceb35:  mov    rbp,rdi
   0x00007f2ff65ceb38:  push   rbx
   0x00007f2ff65ceb39:  sub    rsp,0x48
   0x00007f2ff65ceb3d:  call   0x7f2ff5c3c1d0
=> 0x00007f2ff65ceb42:  vmovdqu xmm0,XMMWORD PTR [rax]
   0x00007f2ff65ceb46:  vpextrq rax,xmm0,0x1
   0x00007f2ff65ceb4c:  vmovdqa XMMWORD PTR [rsp+0x10],xmm0

Setting ARROW_USER_SIMD_LEVEL to None or SSE4_2 didn't help.

Edit: I am using Linux x86-64 with Python 3.14.

Component(s)

Python

[raulcd]

I've been able to replicate and got to this specific function registration arrow::compute::internal::AddSumAvx2AggKernels(arrow::compute::ScalarAggregateFunction*) from the backtrace:

Reading /home/raulcd/code/test-avx/lib/python3.13/site-packages/pyarrow/_compute.cpython-313-x86_64-linux-gnu.so from remote target...                                                                                
[New Thread 70315.70417]
...

Thread 1 received signal SIGILL, Illegal instruction.
0x00007fda73a3cb42 in arrow::compute::internal::AddSumAvx2AggKernels(arrow::compute::ScalarAggregateFunction*) ()
   from target:/home/raulcd/code/test-avx/lib/python3.13/site-packages/pyarrow/libarrow_compute.so.2300

The guard seems correct to me:

arrow/cpp/src/arrow/compute/kernels/aggregate_basic.cc

Lines 1065 to 1069 in 82847d8

	#if defined(ARROW_HAVE_RUNTIME_AVX2)
	if (cpu_info->IsSupported(arrow::internal::CpuInfo::AVX2)) {
	AddSumAvx2AggKernels(func.get());
	}
	#endif

It seems that we pick the flags from /proc/cpuinfo and that contains the host CPU flags not the target so it seems to me that is an issue under qemu.

cc @AntoinePrv @pitrou what are your thoughts?

BTW ARROW_USER_SIMD_LEVEL works for me:

$ qemu-x86_64 -cpu Westmere -L / -E ARROW_USER_SIMD_LEVEL=SSE4_2 $(which python) -c 'import pyarrow._compute; import pyarrow; print(pyarrow.__version__)'
23.0.1

[Alvaro-Kothe]

BTW ARROW_USER_SIMD_LEVEL works for me

Indeed, it works. I had a typo when I first tried it.

[AntoinePrv]

It seems that we pick the flags from /proc/cpuinfo and that contains the host CPU flags not the target

Ho wow, I though we were using CPUID, similar to on Windows.
Fortunately there is ongoing work to fully refactor the whole CpuInfo class.

it seems to me that is an issue under qemu.

I second that diagnosis