With LLVM-14 the Leak Sanitizer crashes after completion of fullstate-simulator-tests

[kuzminrobin]

2022.09.15: Likely to get deprecated after migrating to Rust.

Upon migrating from Clang-13 to Clang-14 some of the test cases of the test src/Qir/Tests/FullstateSimulator/FullstateSimulatorTests.cpp start failing (for pipeline microsoft.qsharp-runtime.sanitized (Build linux) building everything in Debug config):

Original Failure

1: Test command: /home/vsts/work/1/s/src/Qir/Tests/bin/Debug/FullstateSimulator/fullstate-simulator-tests "~[skip]" "-o" "fullstate-simulator-tests_results.xml" "-r" "junit"
1: Working Directory: /home/vsts/work/1/s/src/Qir/Tests/bin/Debug/FullstateSimulator
1: Environment variables: 
1:  LD_LIBRARY_PATH=/home/vsts/work/1/s/build/../src/Simulation/Native/build/drop:/home/vsts/work/1/s/src/Qir/Tests/bin/Debug/bin:/home/vsts/work/1/s/src/Qir/Tests/../Runtime/bin/Debug/bin:
1:  PATH=/home/vsts/work/1/s/build/../src/Simulation/Native/build/drop;/home/vsts/work/1/s/src/Qir/Tests/bin/Debug/bin;/home/vsts/work/1/s/src/Qir/Tests/../Runtime/bin/Debug/bin;
1:  DYLD_LIBRARY_PATH=/home/vsts/work/1/s/build/../src/Simulation/Native/build/drop:/home/vsts/work/1/s/src/Qir/Tests/bin/Debug/bin:/home/vsts/work/1/s/src/Qir/Tests/../Runtime/bin/Debug/bin:
1: Test timeout computed to be: 1500
1: Tracer caught signal 11: addr=0x100000678 pc=0x55b4d242d31a sp=0x7f2eddc00d40
1: ==7283==LeakSanitizer has encountered a fatal error.
1: ==7283==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
1: ==7283==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
1/4 Test #1: fullstate-simulator-tests ........***Failed    0.55 sec
test 2
    Start 2: qir-dynamic-tests

The investigation shows that after the test case completes, the leak sanitizer starts its analysis, gets the signal 11 (SIGSEGV - Signal "Segmentation Violation", typically the null pointer access or an access using a corrupt pointer):

Detailed Failure of a Test Case (see last few lines)

$ LD_LIBRARY_PATH=/mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/build/../src/Simulation/Native/build/drop:/mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/src/Qir/Tests/bin/Debug/bin:/mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/src/Qir/Tests/../Runtime/bin/Debug/bin: /mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/src/Qir/Tests/bin/Debug/FullstateSimulator/fullstate-simulator-tests "Fullstate simulator: allocate qubits"
==7912==Registered root region at 0x7fb0fec007a0 of size 32
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_printf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_sprintf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_snprintf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_fprintf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_vprintf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
==7912==Registered root region at 0x7fb0fe8029f0 of size 176
==7912==Unregistered root region at 0x7fb0fe8029f0 of size 176
==7912==AddressSanitizer: failed to intercept 'xdr_destroy'
==7912==Registered root region at 0x7fb0fe602620 of size 160
==7912==Unregistered root region at 0x7fb0fe602620 of size 160
==7912==AddressSanitizer: failed to intercept 'crypt'
==7912==Registered root region at 0x7fb0fe602620 of size 160
==7912==Unregistered root region at 0x7fb0fe602620 of size 160
==7912==AddressSanitizer: failed to intercept 'crypt_r'
==7912==Registered root region at 0x7fb0fe402dc0 of size 192
==7912==Unregistered root region at 0x7fb0fe402dc0 of size 192
==7912==AddressSanitizer: failed to intercept '__cxa_rethrow_primary_exception'
==7912==AddressSanitizer: libc interceptors initialized
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap  ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000
==7912==Installed the sigaction for signal 11
==7912==Installed the sigaction for signal 7
==7912==Installed the sigaction for signal 8
==7912==T0: stack [0x7fff1bc57000,0x7fff1c457000) size 0x800000; local=0x7fff1c453ed4
==7912==AddressSanitizer Init done
Filters: Fullstate simulator: allocate qubits
==7912==T1: stack [0x7fb0f7e16000,0x7fb0f8614040) size 0x7fe040; local=0x7fb0f8613f44
==7912==T2: stack [0x7fb0f7614000,0x7fb0f7e120c0) size 0x7fe0c0; local=0x7fb0f7e11fc4
==7912==T3: stack [0x7fb0f6e12000,0x7fb0f7610140) size 0x7fe140; local=0x7fb0f7610044
==7912==T4: stack [0x7fb0f6610000,0x7fb0f6e0e1c0) size 0x7fe1c0; local=0x7fb0f6e0e0c4
==7912==T5: stack [0x7fb0f5e0e000,0x7fb0f660c240) size 0x7fe240; local=0x7fb0f660c144
==7912==T6: stack [0x7fb0f560c000,0x7fb0f5e0a2c0) size 0x7fe2c0; local=0x7fb0f5e0a1c4
==7912==T7: stack [0x7fb0f4e0a000,0x7fb0f5608340) size 0x7fe340; local=0x7fb0f5608244
==7912==T8: stack [0x7fb0f4608000,0x7fb0f4e063c0) size 0x7fe3c0; local=0x7fb0f4e062c4
==7912==T9: stack [0x7fb0f3e06000,0x7fb0f4604440) size 0x7fe440; local=0x7fb0f4604344
==7912==T10: stack [0x7fb0f3604000,0x7fb0f3e024c0) size 0x7fe4c0; local=0x7fb0f3e023c4
==7912==T11: stack [0x7fb0f2e02000,0x7fb0f3600540) size 0x7fe540; local=0x7fb0f3600444
===============================================================================
All tests passed (2 assertions in 1 test case)

==7924==Processing thread 7912.
==7924==Stack at 0x7fff1bc57000-0x7fff1c457000 (SP = 0x7fff1c453a68).
==7924==TLS at 0x7fb0ff13d000-0x7fb0ff13f440.
==7924==Processing thread 7913.
==7924==Stack at 0x7fb0f7e16000-0x7fb0f8614040 (SP = 0x7fb0f8613a70).
==7924==TLS at 0x7fb0f8614040-0x7fb0f8616480.
==7924==DTLS 5 at 0x00010000069d-0x00110000179f.
Tracer caught signal 11: addr=0x1000006a0 pc=0x55f9034cf2fa sp=0x7fb0f2e00d40
==7912==LeakSanitizer has encountered a fatal error.
==7912==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==7912==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)

The experiments and the attempts to narrow down the issue didn't clarify the logic behind the failures.
Googling out makes an impression that there is still some unresolved bug in the sanitizer.

Hopefully the issue will be fixed after LLVM-14.0.6.

To Reproduce

# In WSL
# In a clean qsharp-runtime
# In `pwsh`
$Env:BUILD_CONFIGURATION = "Debug"
src/Qir/Runtime/build-qir-runtime.ps1
src/Simulation/Native/build-native-simulator.ps1
src/Qir/Tests/build-qir-tests.ps1

# Either run all tests
src/Qir/Tests/test-qir-tests.ps1
# or in Linux command prompt (not in `pwsh`) run a particular test case (long line is broken up into smaller chunks)
LD_LIBRARY_PATH=\
/mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/build/../src/Simulation/Native/build/drop:\
/mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/src/Qir/Tests/bin/Debug/bin:\
/mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/src/Qir/Tests/../Runtime/bin/Debug/bin:\
 \
/mnt/c/ed/dev/QSharpCompiler/qsharp-runtime/qsharp-runtime_WSL/src/Qir/Tests/bin/Debug/FullstateSimulator/fullstate-simulator-tests \
"Fullstate simulator: allocate qubits"