0.3.34/develop: Zen 4 P/Q override applies to forced coretypes — #6013's dgemm stack overflow still reachable via OPENBLAS_CORETYPE in AVX-512 builds

[iamgahr]

Summary

Since 0.3.34, SGEMM results under a pinned OPENBLAS_CORETYPE (SkylakeX, Haswell) differ between a Zen 4 host and an Intel host for some K, while they are byte-identical with the same pin on every release from 0.3.23 through 0.3.33 (bisected with the conda-forge builds).

The cause is the block added by #5868 at the end of init_parameter() in kernel/setparam-ref.c (v0.3.34 lines 2087–2115): when the host CPUID reports AuthenticAMD, l3_kb % 32768 == 0, l2_kb == 1024 and CPUID.7:EBX bit 16 (AVX512F), it sets sgemm_p=384, sgemm_q=512 (and the d/c/z equivalents). driver/others/dynamic.c reads OPENBLAS_CORETYPE (line 1145), applies force_coretype() (1148) and then calls gotoblas->init() (1180), so the override rewrites the blocking of the forced table on Zen 4/5 hosts only. GEMM_Q is the K-chunk of the reduction, so partial sums are accumulated in a different order than on any other host running the same pinned kernel, and the bits differ. openblas_get_corename() still reports the pinned core, so nothing indicates that the pin no longer means the same thing everywhere.

Reproduction

conda-forge libopenblas 0.3.34 (openmp build), NumPy A @ B in float32, M = N = 64, K varied, OPENBLAS_NUM_THREADS=1. Hosts: AMD EPYC 9V44 (Genoa, Zen 4: L2 1 MiB, L3 32 MiB, AVX-512) vs Intel Xeon Platinum 8573C (Emerald Rapids). Digests of the result matrix compared across the two hosts:

pin	identical K	different K
OPENBLAS_CORETYPE=SkylakeX (table Q = 448)	256, 384, 448, 513, 640, 768, 895, 896	449, 480, 512, 1024, 1536
OPENBLAS_CORETYPE=Haswell (table Q = 320)	256, 320, 513, 600, 639, 640	321, 400, 512, 768, 1024

This is exactly the pattern of the K-chunking rule in driver/level3/level3.c with Q = 512 on the Zen 4 host against the table's Q elsewhere: identical wherever both Q values produce the same chunk sequence (including K = 2Q, where the halving rule coincides), different elsewhere. Predicted from the source before measuring; every K landed as predicted.

Controls, all consistent with the mechanism:

• OPENBLAS_L2_SIZE=2048 on the Zen 4 host (the block requires l2_kb == 1024) restores byte-identity with the Intel host at every K under both pins, with no rebuild. OPENBLAS_L2_SIZE=1024 changes nothing. OPENBLAS_L2_SIZE is inert on the Intel host.
• A Zen 3 host (EPYC 7V13 Milan: L2 512 KiB, no AVX-512) on 0.3.34 is byte-identical to Intel with no workaround, and remains so with OPENBLAS_L2_SIZE=1024 forced (the AVX512F term fails there).
• Every release 0.3.23 … 0.3.33 with the same pins: byte-identical at all K on the same hosts.

Why it matters

OPENBLAS_CORETYPE is documented as a kernel-selection override, but it is the route that has been suggested here for identical results across Intel and AMD (#4510), and up to 0.3.33 it delivered them for GEMM. #6013 found the same block as a crash on Windows; the fix on develop (a59a5c7) only wraps it in #ifndef NO_AVX512, so normal AVX-512 builds of 0.3.35 will keep the host-dependent override under a forced coretype.

Suggestion

Any of: skip the Zen 4 P/Q override when a coretype was forced via OPENBLAS_CORETYPE (so a pin means the same blocking on every host); apply it inside the Zen table's init rather than after force_coretype; or document that OPENBLAS_L2_SIZE=2048 is needed for cross-host identity on Zen 4/5 under a pin. Happy to test a patch on the same hosts.

[iamgahr]

I found this by bisecting across seven machines on two vendors. Would a pre-release check for byte-identity regressions across microarchitectures be useful to you, or do you already have coverage for this? Happy to run master against my fleet and report.

[iamgahr]

This isn't a Windows-specific path — it reproduces on every platform.

The .L12_02b loop in dgemm_kernel_4x8_haswell.S writes 96 bytes per k iteration, unbounded, into a fixed-size frame buffer. The overflow threshold differs only by ABI because the frame sizes differ:

SysV: 256812 + 4096 = 28672 B → overflows at K ≥ 299
Windows: 128812 + 512 = 12800 B → overflows at K ≥ 134

The Windows number is a useful cross-check: @bodono measured the first crash between k=144 and k=160, and the and rsp, -0x1000 stack alignment slack accounts for the 133–176 spread. Deriving the threshold from the source alone lands in the same place.

Reachability (from dynamic.c:913–923; Zen 4 = family 0x19 → exfamily == 10):

NO_AVX512 build, Zen 4/5 → ZEN table → crash. This is #6013, fixed by a59a5c7.
AVX-512 build, Zen 4, no env var → support_avx512_bf16() → COOPERLAKE → dgemm_kernel_16x2_skylakex → safe. (Zen 5 is exfamily 11 → NULL → capability fallback → also SkylakeX/Cooperlake.) This is why there's no mass Linux breakage, and why #5868's authors never saw it.
AVX-512 build + OPENBLAS_CORETYPE=Zen or =Haswell on Zen 4/5 → forced table → the block still fires (#ifndef NO_AVX512 is inert in an AVX-512 build) → dgemm_q=512 → the .S kernel → stack-frame corruption. Not fixed, on any OS.

So case 3 is the one still open. It's not a default-configuration bug post-a59a5c76 — it needs a deliberate opt-in env var plus specific AMD hardware, and it isn't remotely triggerable. That bounds the severity. But it's still live.

Concretely: the reported config in this thread — Linux, conda-forge 0.3.34, Genoa, OPENBLAS_CORETYPE=Haswell — is one DGEMM call away from it. It didn't trigger because KERNEL.HASWELL's SGEMM is sgemm_kernel_8x4_haswell_2.c, a C kernel with no such buffer; only DGEMM routes through the .S one. That's luck, not safety.

[martin-frbg]

Right, that code needs to be guarded by support_avx512() in the DYNAMIC_ARCH case, and maybe it should be opt-in in general, if the changed blocking leads to significantly different results

[iamgahr]

Agreed on the direction, with one correction: support_avx512() is a host test (CPUID.7 + XGETBV), and a Zen 4 host passes it — the block already performs the equivalent check (CPUID.7:EBX bit 16) and fires precisely because the host has AVX-512. So wrapping it in support_avx512() leaves case 3 unchanged: forced Zen or Haswell table on a Zen 4/5 host, host passes both checks, dgemm_q = 512 lands on dgemm_kernel_4x8_haswell.S.

The condition that actually matters is "is the table being initialised an AVX-512 one", and that is known at compile time: in a DYNAMIC_ARCH build setparam-ref.c is compiled once per target, so the target macro is the right guard:

-#ifndef NO_AVX512
+/* Zen 4/5 block sizes are only safe for the AVX-512 tables. Applied to the
+   ZEN/HASWELL fallback tables they hand dgemm_kernel_4x8_haswell.S a k that
+   overruns its fixed panel buffer (28672 B SysV / 12800 B Win, 96 B per k). */
+#if defined(SKYLAKEX) || defined(COOPERLAKE) || defined(SAPPHIRERAPIDS)
 {
     int l3_kb = get_l3_size();

Equivalently, move the block into the existing #if defined(SKYLAKEX) || defined(COOPERLAKE) || defined(SAPPHIRERAPIDS) section of init_parameter() (line 1815 in v0.3.34), which is where per-target tuning already lives. Either form also subsumes the NO_AVX512 guard (no AVX-512 targets are compiled in that case) and adds no symbols, so static builds are unaffected.

I confirmed the kernel half in isolation, with the vendor check out of the picture: v0.3.34, static TARGET=HASWELL, USE_THREAD=0, x86-64 SysV (so the 28672-byte buffer), only change DGEMM_DEFAULT_Q 256 → 512 in the HASWELL section of param.h. DGEMM with kernel n ≥ 24 (the .L12_01 path, the only one that uses the stack sub-buffer) faults from K = 336; the control is clean at every K. That is consistent with the derived threshold: the buffer is exceeded from K ≈ 299, and the overrun reaches the saved return address around 336 — checksums match the control through K = 328, so the frame is corrupted silently before it becomes fatal. At K = 336 rip = 0x0000762800000000 — the return address overwritten with matrix data, the same signature as the dump in #6013. At K = 512 it faults inside the copy loop itself (.L12_02b+18, vmovups %ymm2, 0x20(%rsi)). n < 24 never faults, because .L8_0 has no BUFFER1 reference.

On "significantly different results": no — measured, it is summation-order noise. Same inputs, SGEMM M = N = 64, table Q vs 512:

K	elements differing	max |Δ|	max |C|
449	3715 / 4096	1.7e-5	28
512	3715 / 4096	2.9e-5	34
1024	3748 / 4096	2.6e-5	40
1536	3779 / 4096	3.1e-5	45

That is single-digit ULPs at the output's magnitude — an accuracy non-event. It only matters to someone pinning OPENBLAS_CORETYPE for bit-identity across a mixed fleet, which is the narrow case this issue is about. So I would not make the tuning opt-in for accuracy reasons; the target guard is needed for memory safety regardless. If you want a pin to mean the same blocking on every host as well, the smallest change is to skip the block when getenv("OPENBLAS_CORETYPE") is set — but that is a policy call about what OPENBLAS_CORETYPE promises, and I'm fine either way.

Happy to put up a PR: the target-macro guard (either placement, your call) plus a test that DGEMM at K = 512 under TARGET=HASWELL with DGEMM_DEFAULT_Q = 512 completes and matches the reference — and, if you want the OPENBLAS_CORETYPE skip, that as a separate commit so it can be taken or dropped on its own.

[iamgahr]

Receipts for the reproduction above, so it can be checked rather than taken on trust.

Build. v0.3.34 (e0166008), x86-64 SysV, static, one change:

$ git diff -U1 param.h        # HASWELL section, non-WINDOWS_ABI arm
 #define SGEMM_DEFAULT_Q 320
-#define DGEMM_DEFAULT_Q 256
+#define DGEMM_DEFAULT_Q 512
$ make TARGET=HASWELL BINARY=64 USE_THREAD=0 NO_LAPACK=1 NO_FORTRAN=1 ONLY_CBLAS=1 NO_SHARED=1

Control is the same tree unmodified. Archive contains dgemm_kernel.o built from
dgemm_kernel_4x8_haswell.S (KERNEL.HASWELL). Host: macOS 26.6, Apple clang 17, x86-64
binaries under Rosetta 2 (AVX2 verified). Same ABI, same L_BUFFER_SIZE, same kernel
machine code as Linux; the exact fault K will move with stack layout.

driver (one process per K; exit 0 = survived)

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include "cblas.h"
int main(int argc, char **argv) {
  int K = atoi(argv[1]), M = atoi(argv[2]), N = 64;   /* row-major: kernel n == M */
  double *A = malloc((size_t)M*K*8), *B = malloc((size_t)K*N*8), *C = malloc((size_t)M*N*8);
  uint32_t s = 1;
  for (size_t i = 0; i < (size_t)M*K; i++) { s = s*1103515245u + 12345u; A[i] = (double)(s>>8)/16777216.0 - 0.5; }
  for (size_t i = 0; i < (size_t)K*N; i++) { s = s*1103515245u + 12345u; B[i] = (double)(s>>8)/16777216.0 - 0.5; }
  for (size_t i = 0; i < (size_t)M*N; i++) C[i] = 0.0;
  cblas_dgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, M, N, K, 1.0, A, K, B, N, 0.0, C, N);
  double t = 0; for (size_t i = 0; i < (size_t)M*N; i++) t += C[i];
  printf("ok sum=%.6f\n", t); return 0;
}

Sweep, M = N = 64 (kernel n = 64, .L12_01 path):

K     K*96 B    q=256 (stock)  q=512 (block value)
256   24576     ok             ok
296   28416     ok             ok
298   28608     ok             ok
304   29184     ok             ok
320   30720     ok             ok
328   31488     ok             ok
336   32256     ok             SIGSEGV
344   33024     ok             SIGSEGV
384   36864     ok             SIGSEGV
448   43008     ok             SIGSEGV
512   49152     ok             SIGSEGV

Path dependence, q=512 build, first faulting K in 256…1024 step 8, varying M
(row-major dispatch makes the kernel's n the caller's M):

kernel n    first fault
8           none through 1024
16          none through 1024
23          none through 1024
24          K=336
32          K=336
64          K=336

The flip is exactly at the Ndiv12 = N/24 branch: .L12_01 copies 12 doubles = 96 B per
k into BUFFER1 (three ymm stores at 0x0/0x20/0x40); .L8_0 has no BUFFER1
reference. 28672 / 96 = 298.67, so the overrun begins at K ≥ 299.

Silent band, checksum q=256 vs q=512 below the first fault:

K     q=256                q=512                verdict
296   216.0943529853       216.094353           identical
304   -53.7958054315       -53.795805           identical
312   76.6403060581        76.640306            identical
320   9.9760977432         9.976098             identical
328   -176.3059379524      -176.305938          identical

So the frame is overrun from ~299 and corrupted silently until the overrun reaches the
saved return address; results are still correct in that band.

Fault sites (lldb, q=512 build):

K=336  EXC_BAD_ACCESS   frame #0: 0xffff8e1f40000000
       rip is matrix data — return address overwritten. The value differs run to run
       (an earlier run gave 0x0000762800000000); same signature as the dump in #6013.
K=512  EXC_BAD_ACCESS (address=0x304150000)
       frame #0: .L12_02b + 18    vmovups %ymm2, 0x20(%rsi)
       the copy loop itself walks off the mapped stack.

Not executed here: the composition on Zen 4 hardware — pin Haswell, block fires,
dgemm faults. That follows from the ordering in the report plus the
OPENBLAS_L2_SIZE=2048 control there, which shows the block does fire on that host under
the pin. I'll post the direct run when I'm back on that machine.

[iamgahr]

Ran this on real AMD hardware. It reproduces, and the gate is confirmed rather than assumed.

Host: AWS m8a.2xlarge, AMD EPYC 9R45, family 26 (0x1A) — Zen 5 Turin, not Zen 4
Genoa. Same gate conditions; flagging the difference explicitly. 8 cores, L2 1024 KB/core,
L3 32768 KB, AVX512F + AVX512_BF16. Stock conda-forge libopenblas 0.3.34 (DYNAMIC_ARCH NO_AFFINITY Cooperlake MAX_THREADS=128), numpy 2.4.6, OPENBLAS_NUM_THREADS=1.

The gate (a standalone CPUID probe replicating setparam-ref.c:2095):

vendor           = AuthenticAMD (ebx=0x68747541)   -> PASS
l2_kb            = 1024   (wants == 1024)          -> PASS
l3_kb            = 32768  (wants %32768 == 0)      -> PASS
CPUID.7:EBX b16  = 1      (AVX512F)                -> PASS

ZEN 4/5 BLOCK FIRES ON THIS HOST: YES

Core selected per pin: unpinned → Cooperlake; Haswell → Haswell; Zen → Zen;
SkylakeX → SkylakeX. So the default path is the safe one, as expected.

DGEMM K-sweep, OPENBLAS_CORETYPE=Haswell, M=N=64 (exit code captured directly):

K      K*96 B    result       rc
128    12288     ok           0
256    24576     ok           0
296    28416     ok           0
298    28608     ok           0
304    29184     ok           0
320    30720     ok           0
336    32256     SIGSEGV      139
384    36864     SIGSEGV      139
448    43008     SIGSEGV      139
512    49152     SIGSEGV      139
768    73728     SIGSEGV      139
1024   98304     SIGSEGV      139

First fault at K = 336 — the same K as the isolated x86-64 SysV run I described above,
which had no AMD CPU involved at all. Overrun begins at K ≥ 299 by the buffer arithmetic;
the fault lands at 336 on both.

Controls at K = 512:

OPENBLAS_CORETYPE=Haswell   (block fires)              rc=139  FAULT
OPENBLAS_CORETYPE=Haswell + OPENBLAS_L2_SIZE=2048      rc=0    sum=695.0279288544
OPENBLAS_CORETYPE=Zen       (block fires)              rc=139  FAULT
OPENBLAS_CORETYPE=SkylakeX  (avx512 kernel)            rc=0    sum=695.0279288544
unpinned (default table)                               rc=0    sum=695.0279288544

OPENBLAS_L2_SIZE=2048 makes the gate's l2_kb == 1024 fail, the block does not run, and
the identical call completes. That is the single-variable confirmation that the block is
the cause, not the Haswell kernel per se.

And for the original reproducibility report — same host, same pin, only
OPENBLAS_L2_SIZE differing, so no second machine is needed to isolate the override.
SGEMM, M=N=64, sha256 of the result matrix:

OPENBLAS_CORETYPE=SkylakeX     block ON            block OFF           verdict
256                            629186eabe1e1943    629186eabe1e1943    identical
384                            350e0d7c1f23e89a    350e0d7c1f23e89a    identical
448                            523cedb068cc8b9f    523cedb068cc8b9f    identical
449                            277d196a4246ff1b    2b4f86bb52c56c1d    DIFFERENT
480                            cab9684a185be269    e3b591be27257902    DIFFERENT
512                            2f81b7f2b0976a73    a83d1168fb3a6ec2    DIFFERENT
513                            cef13cf6a7db5311    cef13cf6a7db5311    identical
640                            46a3c90aa490885f    46a3c90aa490885f    identical
768                            f79d1841e120020b    f79d1841e120020b    identical
895                            5ae4a3a455c57e87    5ae4a3a455c57e87    identical
896                            eb3a643ada1e65fa    eb3a643ada1e65fa    identical
1024                           e73887f53fe00a1c    6b4f5b11fa7cddf3    DIFFERENT
1536                           34b90d8cb21ee7d3    9b6266811a1b6d92    DIFFERENT

Differing at exactly K = 449, 480, 512, 1024, 1536 and identical at 256, 384, 448, 513,
640, 768, 895, 896 — the same partition as the table in the original report, and the same
one the GEMM_Q chunking rule in driver/level3/level3.c predicts for Q = 448 vs 512.
The Haswell pin (table Q = 320) gives its own predicted partition, differing from 384
upward and matching at 256, 513, 640.

So all three cases in the table above are now measured: default is safe, and the forced
Zen/Haswell tables both take the override and both fault. #ifndef NO_AVX512 does not
guard this build — the config string shows AVX-512 kernels present.