Clang+Cuda unit test failures

[ndellingwood]

Clang+Cuda builds have some unit test failures (cuda > 10)

clang/15.0.7 + cuda/11.2.2 on Volta70:
graph_cuda

07:06:22 [ RUN      ] Cuda.graph_random_graph_coarsen_double_int_int_TestDevice
07:06:22 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/graph/unit_test/Test_Graph_coarsen.hpp:355: Failure
07:06:22 Value of: correct_coarsening
07:06:22   Actual: false
07:06:22 Expected: true
07:06:22 Coarsening with dedupe method 4 produced invalid coarsening with aggregation heuristic 3.
07:06:22 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/graph/unit_test/Test_Graph_coarsen.hpp:353: Failure
07:06:22 Value of: correct_graph
07:06:22   Actual: false
07:06:22 Expected: true
07:06:22 Coarsening with dedupe method 3 produced invalid graph with aggregation heuristic 5.
07:06:22 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/graph/unit_test/Test_Graph_coarsen.hpp:355: Failure
07:06:22 Value of: correct_coarsening
07:06:22   Actual: false
07:06:22 Expected: true
07:06:22 Coarsening with dedupe method 3 produced invalid coarsening with aggregation heuristic 5.
07:06:22 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/graph/unit_test/Test_Graph_coarsen.hpp:355: Failure
07:06:22 Value of: correct_coarsening
07:06:22   Actual: false
07:06:22 Expected: true
07:06:22 Coarsening with dedupe method 4 produced invalid coarsening with aggregation heuristic 5.
07:06:22 [  FAILED  ] Cuda.graph_random_graph_coarsen_double_int_int_TestDevice (27872 ms)

sparse_cuda

07:31:52 [ RUN      ] Cuda.sparse_spgemm_double_int_int_TestDevice
07:31:52 entries are different.
07:31:52 1 2 7 18 19 24 26 31 35 38 46 51 53 58 63 72 86 96 100 101 ... ... ... 2282 2283 2285 2288 2307 2308 2310 2328 2343 2353 2358 2360 2362 2366 2369 2380 2389 2407 2423 2433 
07:31:52 1 2 7 18 19 24 26 31 35 38 46 51 53 58 63 72 86 96 100 101 ... ... ... 2282 2283 2285 2288 2307 2308 2310 2328 2343 2353 2358 2360 2362 2366 2369 2380 2389 2407 2423 2433 
07:31:52 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/sparse/unit_test/Test_Sparse_spgemm.hpp:308: Failure
07:31:52 Value of: is_identical
07:31:52   Actual: false
07:31:52 Expected: true
07:31:52 SPGEMM_KK
07:31:52 entries are different.
07:31:52 1 2 7 18 19 24 26 31 35 38 46 51 53 58 63 72 86 96 100 101 ... ... ... 2282 2283 2285 2288 2307 2308 2310 2328 2343 2353 2358 2360 2362 2366 2369 2380 2389 2407 2423 2433 
07:31:52 1 2 7 18 19 24 26 31 35 38 46 51 53 58 63 72 86 96 100 101 ... ... ... 2282 2283 2285 2288 2307 2308 2310 2328 2343 2353 2358 2360 2362 2366 2369 2380 2389 2407 2423 2433 
07:31:52 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/sparse/unit_test/Test_Sparse_spgemm.hpp:308: Failure
07:31:52 Value of: is_identical
07:31:52   Actual: false
07:31:52 Expected: true

07:31:52 [ RUN      ] Cuda.sparse_spiluk_double_int_int_TestDevice
# Timeout

blocksparse_cuda

07:43:23 [ RUN      ] Cuda.sparse_block_spgemm_double_int_int_TestDevice
07:43:23 entries are different.
07:43:23 0 1 3 4 5 6 7 9 10 11 12 13 15 16 17 18 19 21 22 24 ... ... ... 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 
07:43:23 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ... ... ... 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 
07:43:23 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/sparse/unit_test/Test_Sparse_bspgemm.hpp:216: Failure
07:43:23 Value of: is_identical
07:43:23   Actual: false
07:43:23 Expected: true
07:43:23 SPGEMM_KK
07:43:23 entries are different.
07:43:23 1 1 2 2 3 4 6 7 8 9 10 11 12 13 14 15 15 16 16 17 ... ... ... 479 480 481 482 483 484 485 486 487 488 489 490 491 492 494 495 496 497 498 499 
07:43:23 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ... ... ... 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 
07:43:23 /home/jenkins/jenkins-new/workspace/KokkosKernels_Nightly_KokkosDev2_CLANG1507_CUDA1122/kokkos-kernels/sparse/unit_test/Test_Sparse_bspgemm.hpp:216: Failure
07:43:23 Value of: is_identical
07:43:23   Actual: false
07:43:23 Expected: true
07:43:23 SPGEMM_KK
07:43:23 [  FAILED  ] Cuda.sparse_block_spgemm_double_int_int_TestDevice (12798 ms)

Reproducer notes (kokkos-dev-2):

source /local-projects/spack/share/spack/setup-env.sh
module use /local-projects/spack-module-aliases/
module purge
module load cmake/3.30.2 cuda/11.2.2 llvm/15.0.7

$KOKKOSKERNELS_PATH/cm_generate_makefile.bash --kokkos-path=$KOKKOS_PATH --with-cuda --compiler=clang++ --arch=Volta70 --cxxstandard=17 --cxxflags="-Wno-unknown-cuda-version -Wall"

make -j16

ctest