nalu-wind +rocm: lld: duplicate symbol: desul::Impl::HIP_SPACE_ATOMIC_LOCKS_DEVICE; defined in nalu-gfx90a.o, SyntheticLidar-gfx90a.o

[eugeneswalker]

Build nalu-wind +rocm amdgpu_target=gfx90a fails using:

• ROCm 5.4.3
• Ubuntu 20.04
• Spack branch exawind-rocm from eugeneswalker/spack fork
  • incorporates changes from @psakievich PR
  • Update exawind packages spack/spack#40793

Spack environment: spack.yaml.txt

Concretization

 -   yszu6dv  exawind@master%[email protected]~cuda~gpu-aware-mpi+hypre~ipo~openfast+rocm~stk_simd~sycl~tiny_profile~umpire amdgpu_target=gfx90a build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  s7oyxte      ^amr-wind@3cb334c76bf462b32e95b97802b18d7b5c097987=main%[email protected]~ascent~cuda~gpu-aware-mpi~hdf5+hypre~ipo~masa+mpi+netcdf~openfast~openmp+rocm+shared~sycl+tests~tiny_profile~umpire amdgpu_target=gfx90a build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  eikfrex          ^hypre@a67b6acc52b263a4aa56b6a7fb09276b78c569b9=develop%[email protected]~caliper~complex~cuda~debug+fortran~gptune~gpu-aware-mpi~int64~internal-superlu~magma~mixedint+mpi~openmp+rocm+shared~superlu-dist~sycl~umpire~unified-memory amdgpu_target=gfx90a build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  4pmvb4g              ^[email protected]%[email protected]~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-ubuntu20.04-x86_64
[+]  opk2fdh                  ^[email protected]%[email protected]+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-ubuntu20.04-x86_64
[+]  bvpocrz                      ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu20.04-x86_64
[+]  x75cg2o                      ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  2gxmxfz                          ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-x86_64
[e]  s7ubsw4              ^[email protected]%[email protected] amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[e]  6xdus6g              ^[email protected]%[email protected]+hiprand amdgpu_target=auto build_system=cmake build_type=Release generator=make patches=a35e689 arch=linux-ubuntu20.04-x86_64
[e]  qq23z2l              ^[email protected]%[email protected]~test amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[e]  kj7ttr7              ^[email protected]%[email protected] amdgpu_target=auto build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[e]  4vf2w2d          ^[email protected]%[email protected]~argobots~cuda+fortran~hwloc+hydra+libxml2+pci~rocm+romio~slurm~vci~verbs~wrapperrpath build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=pmi arch=linux-ubuntu20.04-x86_64
[+]  xab5emm          ^[email protected]%[email protected]+blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize+parallel-netcdf+pic+shared+szip+zstd build_system=autotools patches=0161eb8 arch=linux-ubuntu20.04-x86_64
[+]  rz3egrw              ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=linux-ubuntu20.04-x86_64
[+]  mfkxhgk                  ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  x4ev7kg                      ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  5dtmz2c              ^[email protected]%[email protected]+avx2~ipo build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  np4e7rb                  ^[email protected]%[email protected]+pic build_system=makefile libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  wouiz4z                  ^[email protected]%[email protected]~ipo+pic+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  jrzn4ul              ^[email protected]%[email protected]~cxx~fortran+hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  23vvft6              ^[email protected]%[email protected]~ipo+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  jln5xqc              ^[email protected]%[email protected]~burstbuffer+cxx+fortran+pic+shared build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  2zk6ze6                  ^[email protected]%[email protected]+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu20.04-x86_64
[+]  owwhrha                      ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  ve52wyi              ^[email protected]%[email protected]~programs build_system=makefile libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  itgjywr      ^[email protected]%[email protected]~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-ubuntu20.04-x86_64
[+]  2ybxjtg          ^[email protected]%[email protected]~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-ubuntu20.04-x86_64
[+]  pmltobc              ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  322sfmt              ^[email protected]%[email protected]~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu20.04-x86_64
[+]  thgudgh                  ^ca-certificates-mozilla@2023-05-30%[email protected] build_system=generic arch=linux-ubuntu20.04-x86_64
[+]  x6ksnrs              ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  m7dbp7w          ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  knn3iiz          ^[email protected]%[email protected]+compat+opt build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  w7qwcrr      ^[email protected]%[email protected]~guile build_system=generic arch=linux-ubuntu20.04-x86_64
[e]  vw37kqn      ^[email protected]%[email protected]~cuda+rocm build_system=cmake build_type=Release generator=make patches=5068750,c2ee21c,ca523f1,ddd86f0 arch=linux-ubuntu20.04-x86_64
[e]  xynkkpu      ^[email protected]%[email protected]+image+shared build_system=cmake build_type=Release generator=make patches=9267179 arch=linux-ubuntu20.04-x86_64
[e]  gypfklw      ^[email protected]%[email protected]~link_llvm_dylib~llvm_dylib~openmp+rocm-device-libs build_system=cmake build_type=Release generator=ninja patches=a08bbe1 arch=linux-ubuntu20.04-x86_64
 -   ljsmyht      ^nalu-wind@532e64cd66b3db4f61d501126f437c390dac4d29=main%[email protected]~boost~catalyst~cuda~fftw~gpu-aware-mpi+hypre~ipo~openfast+pic+rocm~shared~tests+tioga+trilinos-solvers~umpire~unit-tests~wind-utils abs_tol=1e-15 amdgpu_target=gfx90a build_system=cmake build_type=Release cxxstd=17 generator=make rel_tol=1e-12 arch=linux-ubuntu20.04-x86_64
[+]  waulukx          ^[email protected]%[email protected]~ipo build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  kpl7elp          ^[email protected]%[email protected]~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings+exodus+explicit_template_instantiation~float+fortran+gtest+hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~panzer~phalanx~piro~python+rocm~rocm_rdc~rol~rythmos+sacado~scorec+shards~shared~shylu+stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~test~thyra+tpetra~trilinoscouplings~wrapper~x11+zoltan~zoltan2 amdgpu_target=gfx90a build_system=cmake build_type=Release cxxstd=17 generator=make gotype=long_long arch=linux-ubuntu20.04-x86_64
[+]  q3r7xh3              ^[email protected]%[email protected]~atomic~chrono~clanglibcpp~container~context~contract~coroutine~date_time~debug+exception~fiber~filesystem+graph~graph_parallel~icu~iostreams~json~locale~log+math~mpi+multithreaded~nowide~numpy~pic~program_options~python~random~regex~serialization+shared~signals~singlethreaded+stacktrace~system~taggedlayout~test~thread~timer~type_erasure~versionedlayout~wave build_system=generic cxxstd=98 patches=a440f96 visibility=hidden arch=linux-ubuntu20.04-x86_64
[+]  gp3lu6n              ^[email protected]%[email protected]~base_scope~fortran+hdf5~int64~ipo~legacy~mem_debug+mpi~pic+scoping+shared~static~testing~tools build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  d4hamal              ^[email protected]%[email protected]~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  47tgik5                  ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  qp7l7qc                      ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  26jc2qb                      ^[email protected]%[email protected] build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  tj5lldx                  ^[email protected]%[email protected]+pic~python+shared build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  naalujk                      ^[email protected]%[email protected]~pic build_system=autotools libs=shared,static arch=linux-ubuntu20.04-x86_64
[+]  3rygj2u              ^[email protected]%[email protected]+hdf5+shared+zlib build_system=autotools arch=linux-ubuntu20.04-x86_64
[+]  q5mrfhv              ^[email protected]%[email protected]~gdb~int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903,b1225da arch=linux-ubuntu20.04-x86_64
[+]  szx3vir              ^[email protected]%[email protected]~gdb~int64~ipo+shared build_system=cmake build_type=Release generator=make patches=4f89253,50ed208,704b84f arch=linux-ubuntu20.04-x86_64
[+]  p2zzsws      ^tioga@73a690616d4ca6478a3968fb6b6de3acd92bc71b=develop%[email protected]~ipo~nodegid+pic+shared~stats~timers build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64
[+]  iktbvkr      ^[email protected]%[email protected]~ipo+pic+shared~tests build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-x86_64

Using Docker container image w/ pre-installed ROCm 5.4.3

• ecpe4s/ubuntu20.04-runner-amd64-gcc-11.4-rocm5.4.3-mpi-base:2023.08.20

$> docker run -it ecpe4s/ubuntu20.04-runner-amd64-gcc-11.4-rocm5.4.3-mpi-base:2023.08.20
root@de172ee709b6:/# git clone https://github.com/eugeneswalker/spack /spack
root@de172ee709b6:/# git -C /spack checkout exawind-rocm
root@de172ee709b6:/# . /spack/share/spack/setup-env.sh
root@de172ee709b6:/# spack -e . concretize -f -j24 | tee concretize.log
root@de172ee709b6:/# spack -e . install -j24
...
==> Installing nalu-wind-532e64cd66b3db4f61d501126f437c390dac4d29=main-ljsmyht6i77rhfbu3zpysac3gfmuhkcu [43/44]
==> No binary for nalu-wind-532e64cd66b3db4f61d501126f437c390dac4d29=main-ljsmyht6i77rhfbu3zpysac3gfmuhkcu found: installing from source
==> No patches needed for nalu-wind
==> nalu-wind: Executing phase: 'cmake'
==> nalu-wind: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j24'
...
     926     [100%] Linking CXX executable naluX
     927     /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/cmake-3.27.7-itgjywrzoenlycn6gkzxhoip7vw7cnpy/bin/cmake -E cmake_link_script CMakeFiles/naluX.dir/link.txt --verbose=1
     928     /opt/rocm-5.4.3/bin/hipcc -fgpu-rdc -O3 -DNDEBUG -Wl,-rpath -Wl,/usr/local/mpich/install/mpich/lib CMakeFiles/naluX.dir/nalu.C.o -o naluX  -Wl,-rpath,/spack/opt/spack/linux-ubuntu2
             0.04-x86_64/gcc-11.4.0/cgns-4.4.0-gp3lu6ngi7tpkfuo6vnprustq2egrcis/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/netcdf-c-4.9.2-xab5emmn33znxhkjmbm77levizx4jj3v/lib:/spa
             ck/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/parallel-netcdf-1.12.3-jln5xqcszsaep4uclxoj5ibaaxfep3j3/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/hdf5-1.14.3-jrzn4u
             l7kubphk2fjauqci5yv3nnjg43/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/parmetis-4.0.3-szx3virtswng6jbmwcfsrz3zbibvlby7/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gc
             c-11.4.0/metis-5.1.0-q5mrfhv5itrdu4oj22eoyaznodwadaz2/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/zlib-ng-2.1.4-knn3iizbugamvvawnao4crtzm7vtp2q4/lib:/spack/opt/spack/l
             inux-ubuntu20.04-x86_64/gcc-11.4.0/openblas-0.3.24-4pmvb4gfgtcv4buit3j7pkuhadivloso/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyv
             ngqgpa/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/hwloc-2.9.1-d4hamal53fekiz2vrmtofwolw4t3fyld/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/yaml-cpp-0.7.0
             -iktbvkrf23aa7vvjgogvsuzwp6gbw667/lib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/hypre-a67b6acc52b263a4aa56b6a7fb09276b78c569b9_develop-eikfrexb5rcibaiiuowaws7uf2cnqcvz/l
             ib:/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/tioga-73a690616d4ca6478a3968fb6b6de3acd92bc71b_develop-p2zzswsitdab7p5cv4h4htz3bvoluoyz/lib:/usr/local/mpich/install/mpich/l
             ib::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
             :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: libnalu.a /opt/rocm-5.4.3/lib/libamdhip64.so.5.4.50403 /opt/rocm-5.4.3/llvm/lib/clang/15.0.0/lib/linux/li
             bclang_rt.builtins-x86_64.a --hip-link --offload-arch=gfx90a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/li
             bmuelu-adapters.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libmuelu.a /spack/opt/spack/linux-ubuntu20.04
             -x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_emend.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kp
             l7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_expreval.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_sea
             rch_util_base.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_unit_test_utils.a /spack/opt/spack/linux
             -ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_mesh_fixtures.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-
             13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_balance_lib.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgl
             tfxkxln4/lib/libstk_tools_lib.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_transfer_utils_lib.a /sp
             ack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_transfer_impl.a /spack/opt/spack/linux-ubuntu20.04-x86_64/g
             cc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_search.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgr
             grnk4phoad6zgltfxkxln4/lib/libstk_io_util.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_io.a /spack/
             opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_mesh_base.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.
             0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_topology.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4p
             hoad6zgltfxkxln4/lib/libstk_ngp_test.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_math.a /spack/opt
             /spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_coupling.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/tr
             ilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_util_command_line.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrg
             rnk4phoad6zgltfxkxln4/lib/libstk_util_diag.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_util_regist
             ry.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_util_env.a /spack/opt/spack/linux-ubuntu20.04-x86_6
             4/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_util_parallel.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-
             kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libstk_util_util.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libifpa
             ck2.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libanasazitpetra.a /spack/opt/spack/linux-ubuntu20.04-x86
             _64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libModeLaplace.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7
             elpqgrgrnk4phoad6zgltfxkxln4/lib/libanasaziepetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libanasazi.
             a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libmapvarlib.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-
             11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libsuplib_cpp.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrn
             k4phoad6zgltfxkxln4/lib/libsuplib_c.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libsuplib.a /spack/opt/sp
             ack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libsupes.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.
             4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libaprepro_lib.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln
             4/lib/libio_info_lib.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIonit.a /spack/opt/spack/linux-ubuntu
             20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIotr.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kp
             l7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIohb.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIogs.a /spack/o
             pt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIotm.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos
             -13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIogn.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/l
             ib/libIovs.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIocgns.a /spack/opt/spack/linux-ubuntu20.04-x86
             _64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libIoex.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrg
             rnk4phoad6zgltfxkxln4/lib/libIoss.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/cgns-4.4.0-gp3lu6ngi7tpkfuo6vnprustq2egrcis/lib/libcgns.so /spack/opt/spack/linux-ubuntu20.
             04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libnemesis.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kp
             l7elpqgrgrnk4phoad6zgltfxkxln4/lib/libexoIIv2for32.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libexodus_
             for.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libexodus.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc
             -11.4.0/netcdf-c-4.9.2-xab5emmn33znxhkjmbm77levizx4jj3v/lib/libnetcdf.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/parallel-netcdf-1.12.3-jln5xqcszsaep4uclxoj5ibaaxfep3j
             3/lib/libpnetcdf.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libamesos2.a /spack/opt/spack/linux-ubuntu2
             0.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libbelosxpetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02
             .28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libbelostpetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libbe
             losepetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libbelos.a /spack/opt/spack/linux-ubuntu20.04-x86_6
             4/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libml.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4
             phoad6zgltfxkxln4/lib/libifpack.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libzoltan2.a /spack/opt/spack
             /linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libamesos.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.
             0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libgaleri-xpetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln
             4/lib/libgaleri-epetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libaztecoo.a /spack/opt/spack/linux-ub
             untu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libxpetra-sup.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.202
             3.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libxpetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libtri
             linosss.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libtpetraext.a /spack/opt/spack/linux-ubuntu20.04-x86
             _64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libtpetrainout.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7
             elpqgrgrnk4phoad6zgltfxkxln4/lib/libtpetra.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libtpetraclassic.a
              /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libkokkostsqr.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-
             11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libepetraext.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/hdf5-1.14.3-jrzn4ul7kubphk2fjauqci5yv3nnj
             g43/lib/libhdf5.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libtriutils.a /spack/opt/spack/linux-ubuntu2
             0.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libshards.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-k
             pl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libzoltan.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/parmetis-4.0.3-szx3virtswng6jbmwcfsrz3zbibvlby7/lib/libparmetis.so /spack/opt/s
             pack/linux-ubuntu20.04-x86_64/gcc-11.4.0/metis-5.1.0-q5mrfhv5itrdu4oj22eoyaznodwadaz2/lib/libmetis.so -lm /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/zlib-ng-2.1.4-knn3iiz
             bugamvvawnao4crtzm7vtp2q4/lib/libz.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libepetra.a /spack/opt/sp
             ack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libsacado.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13
             .4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libkokkoskernels.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/metis-5.1.0-q5mrfhv5itrdu4oj22eoyaznodwadaz2/lib/libmet
             is.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libteuchoskokkoscomm.a /spack/opt/spack/linux-ubuntu20.04
             -x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libteuchoskokkoscompat.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.202
             3.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libteuchosremainder.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/
             lib/libteuchosnumerics.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/openblas-0.3.24-4pmvb4gfgtcv4buit3j7pkuhadivloso/lib/libopenblas.so /spack/opt/spack/linux-ubuntu20.04
             -x86_64/gcc-11.4.0/openblas-0.3.24-4pmvb4gfgtcv4buit3j7pkuhadivloso/lib/libopenblas.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrn
             k4phoad6zgltfxkxln4/lib/libteuchoscomm.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libteuchosparameterlis
             t.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libteuchosparser.a /spack/opt/spack/linux-ubuntu20.04-x86_6
             4/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libteuchoscore.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf
             3xloyvngqgpa/lib/libboost_graph.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_graph-mt.so /spack/opt/spack/linu
             x-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_stacktrace_noop-mt.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q
             3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_stacktrace_basic.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_sta
             cktrace_noop.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_stacktrace_addr2line.so /spack/opt/spack/linux-ubunt
             u20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_stacktrace_addr2line-mt.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r
             7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_stacktrace_basic-mt.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_st
             acktrace_backtrace.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/boost-1.83.0-q3r7xh3tyrlmze67zrwf3xloyvngqgpa/lib/libboost_stacktrace_backtrace-mt.so /spack/opt/spack/li
             nux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libkokkossimd.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4
             .0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libkokkosalgorithms.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfx
             kxln4/lib/libkokkoscontainers.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libkokkoscore.a /usr/lib/x86_64
             -linux-gnu/libdl.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/hwloc-2.9.1-d4hamal53fekiz2vrmtofwolw4t3fyld/lib/libhwloc.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-
             11.4.0/trilinos-13.4.0.2023.02.28-kpl7elpqgrgrnk4phoad6zgltfxkxln4/lib/libgtest.a /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/yaml-cpp-0.7.0-iktbvkrf23aa7vvjgogvsuzwp6gbw6
             67/lib/libyaml-cpp.so.0.7.0 /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/hypre-a67b6acc52b263a4aa56b6a7fb09276b78c569b9_develop-eikfrexb5rcibaiiuowaws7uf2cnqcvz/lib/libHYPR
             E.so /spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-11.4.0/tioga-73a690616d4ca6478a3968fb6b6de3acd92bc71b_develop-p2zzswsitdab7p5cv4h4htz3bvoluoyz/lib/libtioga.so /usr/local/mpich/i
             nstall/mpich/lib/libmpicxx.so /usr/local/mpich/install/mpich/lib/libmpi.so -ldl
  >> 929     lld: error: duplicate symbol: desul::Impl::HIP_SPACE_ATOMIC_LOCKS_DEVICE
     930     >>> defined in /tmp/nalu-5bc2fe/nalu-gfx90a.o
     931     >>> defined in /tmp/SyntheticLidar-f4e425/SyntheticLidar-gfx90a.o
     932
  >> 933     lld: error: duplicate symbol: desul::Impl::HIP_SPACE_ATOMIC_LOCKS_NODE
     934     >>> defined in /tmp/nalu-5bc2fe/nalu-gfx90a.o
     935     >>> defined in /tmp/SyntheticLidar-f4e425/SyntheticLidar-gfx90a.o
     936
  >> 937     lld: error: duplicate symbol: Kokkos::Impl::g_device_hip_lock_arrays
     938     >>> defined in /tmp/nalu-5bc2fe/nalu-gfx90a.o
     939     >>> defined in /tmp/SyntheticLidar-f4e425/SyntheticLidar-gfx90a.o
...
  >> 1009    lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
  >> 1010    clang-15: error: amdgcn-link command failed with exit code 1 (use -v to see invocation)
  >> 1011    make[2]: *** [CMakeFiles/naluX.dir/build.make:221: naluX] Error 1
     1012    make[2]: Leaving directory '/tmp/root/spack-stage/spack-stage-nalu-wind-532e64cd66b3db4f61d501126f437c390dac4d29_main-ljsmyht6i77rhfbu3zpysac3gfmuhkcu/spack-build-ljsmyht'
  >> 1013    make[1]: *** [CMakeFiles/Makefile2:423: CMakeFiles/naluX.dir/all] Error 2
     1014    make[1]: Leaving directory '/tmp/root/spack-stage/spack-stage-nalu-wind-532e64cd66b3db4f61d501126f437c390dac4d29_main-ljsmyht6i77rhfbu3zpysac3gfmuhkcu/spack-build-ljsmyht'
  >> 1015    make: *** [Makefile:139: all] Error 2