[Python] Locally built wheels fail python tests

[boschmitt]

Required prerequisites

• Consult the security policy. If reporting a security vulnerability, do not report the bug using this form. Use the process described in the policy to report the issue.
• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker to verify that this hasn't already been reported. +1 or comment there if it has.
• If possible, make a PR with a failing test to give us a starting point to work on!

Describe the bug

Likely to be relate to: #2347

When trying to test locally built wheels, I run into the following problem:

root@9948b97c1984:/cuda-quantum# pytest python/tests/
============================================= test session starts =============================================
platform linux -- Python 3.10.12, pytest-8.2.0, pluggy-1.5.0
rootdir: /cuda-quantum
configfile: pyproject.toml
plugins: anyio-4.6.2.post1
collected 768 items / 8 skipped

python/tests/backends/test_Quantinuum_LocalEmulation_builder.py .......                                 [  0%]
python/tests/backends/test_Quantinuum_LocalEmulation_kernel.py ............                             [  2%]
python/tests/backends/test_braket.py ssssssssssssssssssssss                                             [  5%]
python/tests/backends/test_stim.py .....                                                                [  5%]
python/tests/builder/test_NoiseModel.py .F.F..F.F.....F.....                                            [  8%]
python/tests/builder/test_SpinOperator.py ..............                                                [ 10%]
python/tests/builder/test_cupy_integration.py ........                                                  [ 11%]
python/tests/builder/test_kernel_builder.py ................................................            [ 17%]
python/tests/builder/test_observe.py ................................                                   [ 21%]
python/tests/builder/test_optimizer.py ..............................................                   [ 27%]
python/tests/builder/test_qalloc_init.py .........F.............................                        [ 32%]
python/tests/builder/test_qalloc_init_state.py ...................                                      [ 35%]
python/tests/builder/test_qpp_target.py .                                                               [ 35%]
python/tests/builder/test_quake_value.py .....                                                          [ 36%]
python/tests/builder/test_sample.py ...........................sssssssss............                    [ 42%]
python/tests/builder/test_state.py .....                                                                [ 43%]
python/tests/builder/test_vqe.py ..........................................................             [ 50%]
python/tests/custom/test_custom_operations.py ................                                          [ 52%]
python/tests/custom/test_euler_decomposition.py ...                                                     [ 53%]
python/tests/custom/test_kak_decomposition.py ..                                                        [ 53%]
python/tests/domains/test_chemistry.py ......                                                           [ 54%]
python/tests/domains/test_qnn.py ..                                                                     [ 54%]
python/tests/handlers/test_photonics_kernel.py .......s..                                               [ 55%]
python/tests/kernel/test_adjoint_operations.py ........                                                 [ 56%]
python/tests/kernel/test_control_operations.py ..........                                               [ 58%]
python/tests/kernel/test_ir_operations.py ...                                                           [ 58%]
python/tests/kernel/test_kernel_call_return.py ...                                                      [ 58%]
python/tests/kernel/test_kernel_complex.py .................                                            [ 61%]
python/tests/kernel/test_kernel_exp_pauli.py ..                                                         [ 61%]
python/tests/kernel/test_kernel_features.py ........................................................... [ 69%]
.............                                                                                           [ 70%]
python/tests/kernel/test_kernel_float.py ...............                                                [ 72%]
python/tests/kernel/test_kernel_qvector_init.py ..........................                              [ 76%]
python/tests/kernel/test_kernel_qvector_state_init.py ......................                            [ 78%]
python/tests/kernel/test_kernel_qview.py ..                                                             [ 79%]
python/tests/kernel/test_kernel_return.py .................                                             [ 81%]
python/tests/kernel/test_kernel_translate.py .................                                          [ 83%]
python/tests/kernel/test_library_kernels.py .                                                           [ 83%]
python/tests/kernel/test_observe_kernel.py .........                                                    [ 84%]
python/tests/kernel/test_sample_kernel.py .....                                                         [ 85%]
python/tests/kernel/test_state_kernel.py .....                                                          [ 86%]
python/tests/kernel/test_state_mps.py ....                                                              [ 86%]
python/tests/kernel/test_trotter.py ..                                                                  [ 86%]
python/tests/kernel/test_vqe_kernel.py .....                                                            [ 87%]
python/tests/mlir/test_output_qir.py .                                                                  [ 87%]
python/tests/mlir/test_output_translate_openqasm.py .                                                   [ 87%]
python/tests/mlir/test_output_translate_qir.py .                                                        [ 88%]
python/tests/operator/integrators/test_evolve_dynamics_torch_integrators.py FFFFFF                      [ 88%]
python/tests/operator/test_evolve_dynamics.py .F.F.F.F.FFF                                              [ 90%]
python/tests/operator/test_evolve_simulators.py ..                                                      [ 90%]
python/tests/operator/test_operator.py ....................                                             [ 93%]
python/tests/parallel/test_mpi_api.py s                                                                 [ 93%]
python/tests/parallel/test_mpi_mqpu.py ss                                                               [ 93%]
python/tests/parallel/test_mqpu.py ........                                                             [ 94%]
python/tests/remote/test_remote_code_exec.py .Fatal Python error: Segmentation fault

Current thread 0x000075bf256f8480 (most recent call first):
  File "/cuda-quantum/python/tests/remote/test_remote_code_exec.py", line 152 in test_optimizer
  File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 162 in pytest_pyfunc_call
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/python.py", line 1627 in runtest
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 241 in <lambda>
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 341 in from_call
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 240 in call_and_report
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/usr/local/lib/python3.10/dist-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 339 in _main
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 285 in wrap_session
  File "/usr/local/lib/python3.10/dist-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_callers.py", line 103 in _multicall
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/usr/local/lib/python3.10/dist-packages/pluggy/_hooks.py", line 513 in __call__
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 178 in main
  File "/usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/usr/local/bin/pytest", line 8 in <module>

Extension modules: markupsafe._speedups, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, cuquantum.bindings._utils, cuquantum.bindings._internal.cudensitymat, cuquantum.bindings.cycudensitymat, cuquantum.bindings.cudensitymat, cuquantum._utils, cuquantum.custatevec._internal.custatevec, cuquantum.custatevec.cycustatevec, cuquantum.custatevec.custatevec, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, PIL._imaging, kiwisolver._cext, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._direct, cupy_backends.cuda._softlink, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy._util, cupy.cuda.device, fastrlock.rlock, cupy.cuda.memory_hook, cupy_backends.cuda.stream, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc, cupy.cuda.pinned_memory, cupy.cuda.common, cupy.cuda.cub, cupy_backends.cuda.libs.nvtx, cupy.cuda.thrust, cupy._core._dtype, cupy._core._scalar, cupy._core._accelerator, cupy._core._memory_range, cupy_backends.cuda.libs.cutensor, cupy._core._fusion_thread_local, cupy._core._kernel, cupy._core._routines_manipulation, cupy._core._routines_binary, cupy._core._optimize_config, cupy._core._cub_reduction, cupy._core._reduction, cupy._core._routines_math, cupy._core._routines_indexing, cupy._core._routines_linalg, cupy._core._routines_logic, cupy._core._routines_sorting, cupy._core._routines_statistics, cupy._core.dlpack, cupy._core.flags, cupy._core.core, cupy._core._fusion_variable, cupy._core._fusion_trace, cupy._core._fusion_kernel, cupy._core.new_fusion, cupy._core.fusion, cupy._core.raw, cupy.fft._cache, cupy.fft._callback, cupy.random._bit_generator, cupyx.cutensor, scipy._lib._uarray._uarray, cupy.lib._polynomial, cuquantum.cutensornet._internal.cutensornet, cuquantum.cutensornet.cycutensornet, cuquantum.cutensornet.cutensornet, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, cupy_backends.cuda.libs.cublas, charset_normalizer.md, psutil._psutil_linux, psutil._psutil_posix, cupy.random._generator_api, cupy_backends.cuda.libs.cusolver (total: 214)
Segmentation fault (core dumped)

The remote code execution tests, which segfaulted in the previous complete tests run, seems to pass when run alone, as well as with together with other remote tests:

root@9948b97c1984:/cuda-quantum# pytest python/tests/remote/
============================================= test session starts =============================================
platform linux -- Python 3.10.12, pytest-8.2.0, pluggy-1.5.0
rootdir: /cuda-quantum
configfile: pyproject.toml
plugins: anyio-4.6.2.post1
collected 39 items

python/tests/remote/test_remote_code_exec.py .........s.........s                                       [ 51%]
python/tests/remote/test_remote_platform.py ...................                                         [100%]

======================================= 37 passed, 2 skipped in 12.55s ========================================

The same is true for the builder noise model test:

root@9948b97c1984:/cuda-quantum# pytest python/tests/builder/
============================================= test session starts ==============================================
platform linux -- Python 3.10.12, pytest-8.2.0, pluggy-1.5.0
rootdir: /cuda-quantum
configfile: pyproject.toml
plugins: anyio-4.6.2.post1
collected 343 items

python/tests/builder/test_NoiseModel.py ....................                                             [  5%]
python/tests/builder/test_SpinOperator.py ..............                                                 [  9%]
python/tests/builder/test_cupy_integration.py ........                                                   [ 12%]
python/tests/builder/test_kernel_builder.py ................................................             [ 26%]
python/tests/builder/test_observe.py ................................                                    [ 35%]
python/tests/builder/test_optimizer.py ..............................................                    [ 48%]
python/tests/builder/test_qalloc_init.py .........F.............................                         [ 60%]
python/tests/builder/test_qalloc_init_state.py ...................                                       [ 65%]
python/tests/builder/test_qpp_target.py .                                                                [ 66%]
python/tests/builder/test_quake_value.py .....                                                           [ 67%]
python/tests/builder/test_sample.py ...........................sssssssss............                     [ 81%]
python/tests/builder/test_state.py .....                                                                 [ 83%]
python/tests/builder/test_vqe.py ..........................................................              [100%]

=================================================== FAILURES ===================================================
________________________________________ test_kernel_complex_force_kron ________________________________________

    @skipIfNvidiaFP64NotInstalled
    def test_kernel_complex_force_kron():
        cudaq.reset_target()
        cudaq.set_target('nvidia-fp64')

        c = [0. + 0j] * 1024
        c[1023] = 1j

        kernel, vec = cudaq.make_kernel(list[complex])
        p = kernel.qalloc(1)
        q = kernel.qalloc(vec)
        kernel.mz(p)
        kernel.mz(q)

        counts = cudaq.sample(kernel, c)
>       assert len(counts) == 1
E       assert 51 == 1
E        +  where 51 = len(<cudaq.mlir._mlir_libs._quakeDialects.cudaq_runtime.SampleResult object at 0x7497b5da0bf0>)

python/tests/builder/test_qalloc_init.py:189: AssertionError
=========================================== short test summary info ============================================
FAILED python/tests/builder/test_qalloc_init.py::test_kernel_complex_force_kron - assert 51 == 1
================================== 1 failed, 333 passed, 9 skipped in 17.28s ===================================
is target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.
[warning] Target nvidia-fp64: This target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.
[warning] Target nvidia-fp64: This target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.
[warning] Target nvidia-fp64: This target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.
[warning] Target nvidia-fp64: This target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.
[warning] Target nvidia-fp64: This target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.
[warning] Target nvidia-fp64: This target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.
{ 00:481 11:519 }
{ 1:10 }
{ 11111:10 }
{ 111111111:10 }
{ 1:100 }
{ 11111:100 }
{ 111111111:100 }
{ 1:1000 }
{ 11111:1000 }
{ 111111111:1000 }
{ 111:10 }
{ 11111:10 }
{ 111111111:10 }
{ 111:100 }
{ 11111:100 }
{ 111111111:100 }
{ 111:1000 }
{ 11111:1000 }
{ 111111111:1000 }
{ 111:10 }
{ 11111:10 }
{ 111111111:10 }
{ 111:100 }
{ 11111:100 }
{ 111111111:100 }
{ 111:1000 }
{ 11111:1000 }
{ 111111111:1000 }
{
  IIZ : { 1:10 }
   IZI : { 1:10 }
   ZII : { 1:10 }
   __global__ : { }
}
{
  IIZ : { 1:10 }
   IZI : { 1:10 }
   ZII : { 1:10 }
   __global__ : { }
}
{
  IIZ : { 1:100 }
   IZI : { 1:100 }
   ZII : { 1:100 }
   __global__ : { }
}
{
  IIZ : { 1:100 }
   IZI : { 1:100 }
   ZII : { 1:100 }
   __global__ : { }
}
{ 00000:510 11111:490 }

Although test_qalloc_init.py fails in this case, which also passes when run alone:

root@9948b97c1984:/cuda-quantum# pytest python/tests/builder/test_qalloc_init.py
=================================================== test session starts ====================================================
platform linux -- Python 3.10.12, pytest-8.2.0, pluggy-1.5.0
rootdir: /cuda-quantum
configfile: pyproject.toml
plugins: anyio-4.6.2.post1
collected 39 items

python/tests/builder/test_qalloc_init.py .......................................                                     [100%]

==================================================== 39 passed in 2.18s ====================================================
Q documentation for more information.
[warning] Target nvidia-fp64: This target is deprecating. Please use the 'nvidia' target with option 'fp64' by adding the command line option '--target-option fp64' or passing it as cudaq.set_target('nvidia', option='fp64') in Python. Please refer to CUDA-Q documentation for more information.

Steps to reproduce the bug

docker pull ghcr.io/nvidia/cuda-quantum-devdeps:ext-cu12.0-gcc11-main`
docker run -d --name cudaq-test --gpus all ghcr.io/nvidia/cuda-quantum-devdeps:ext-cu12.0-gcc11-main tail -f /dev/null
docker exec -it cudaq-test /bin/bash

git clone https://github.com/NVIDIA/cuda-quantum.git
apt update && apt install python3.10-venv
pip3 install build
cd cudaq-quantum
python3 -m build --wheel
pip install dist/cuda_quantum_cu12-0.0.0-cp310-cp310-linux_x86_64.whl
pytest python/tests/

Expected behavior

Tests should pass.

Is this a regression? If it is, put the last known working version (or commit) here.

Not a regression

Environment

• CUDA-Q version: main (5435af0)
• Python version: 3.10
• C++ compiler: gcc11 ? (what is in the devdeps image)
• Operating system: Latest ghcr.io/nvidia/cuda-quantum-devdeps:ext-cu12.0-gcc11-main

Suggestions

It seems that global some global state is not being properly cleaned up so tests will fail depending on which other test run before.