matchedNodeElmReader alloc crash

[KennethEJansen]

Sorry if this has been covered before but how big of a mesh should I be able to stream in with this build settingkjansen@pfe26:~/SCOREC-core/buildMGEN_write3D> more doConfigure14_18
#!/bin/bash -ex

For Chef

cmake
-DCMAKE_C_COMPILER=mpicc
-DCMAKE_CXX_COMPILER=mpicxx
-DSCOREC_CXX_WARNINGS=OFF
-DSCOREC_CXX_OPTIMIZE=ON
-DSCOREC_CXX_SYMBOLS=ON
-DENABLE_ZOLTAN=ON
-DENABLE_SIMMETRIX=ON
-DPCU_COMPRESS=ON
-DSIM_MPI="mpt"
-DSIM_PARASOLID=ON
-DMDS_SET_MAX=1024
-DMDS_ID_TYPE=long
-DIS_TESTING=ON
-DMESHES=/projects/tools/SCOREC-core/meshes
-DCMAKE_INSTALL_PREFIX=$PWD/install
../core

#-DMDS_ID_TYPE=int or long

I have progressively thrown more nodes at it (doubling compute nodes 4 times now even though the number of mesh nodes only doubled from a prior successful run) but keep getting this stack trace

MPT: #7 0x00002aaab019c61a in abort () from /lib64/libc.so.6
MPT: #8 0x000000000049109b in reel_fail (
MPT: format=format@entry=0x49be6e "realloc(%p, %lu) failed")
MPT: at /home5/kjansen/SCOREC-core/core/pcu/reel/reel.c:24
MPT: #9 0x0000000000490fc8 in noto_realloc (p=0x5a756a90,
MPT: size=18446744056901788296)
MPT: at /home5/kjansen/SCOREC-core/core/pcu/noto/noto_malloc.c:60
MPT: #10 0x0000000000490626 in pcu_push_buffer (b=0x3afd84b8,
MPT: size=size@entry=18446744056901788288)
MPT: at /home5/kjansen/SCOREC-core/core/pcu/pcu_buffer.c:37
MPT: #11 0x0000000000490917 in pcu_msg_pack (m=m@entry=0x6b79c0 <global_pmsg>,
MPT: id=id@entry=639, size=size@entry=18446744056901788288)
MPT: at /home5/kjansen/SCOREC-core/core/pcu/pcu_msg.c:133
MPT: #12 0x000000000048eb5f in PCU_Comm_Pack (to_rank=to_rank@entry=639,
MPT: data=data@entry=0x2357cb30, size=18446744056901788288)
MPT: at /home5/kjansen/SCOREC-core/core/pcu/pcu.c:141
MPT: #13 0x000000000047e733 in apf::setCoords (m=m@entry=0x3312d300,
MPT: coords=0x2357cb30, nverts=3015292, globalToVert=...)
MPT: at /home5/kjansen/SCOREC-core/core/apf/apfConstruct.cc:202
MPT: #14 0x000000000043722c in main (argc=, argv=)
MPT: at /home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc:832

[KennethEJansen]

I seem to be crashing when trying to stream in a 8,437,865,894 element, 1.9B node mesh. It is a mix of wedges and tets. I was trying to stream it in to 160 Broadwell nodes each running 4 processes and I think that since I told PBS this:
#PBS -l select=160:ncpus=4:mpiprocs=4:model=bro
it should have given each process 192G/4= 48G so that is is 13.2M elements and 3M nodes per process which is better than what I was last successful with (half the mesh sizes in both elements and nodes ran through on 40 nodes with 1 process per core). I guess this means it is either an index limit or a balloon of memory usage with more processes or PBS not really doing what it should in distributing the processes (though I have sshed to a node and found the expected 4 processes running there).

[KennethEJansen]

If I am reading the output right,

MPT: #13 0x000000000047e733 in apf::setCoords (m=m@entry=0x3312d300,
MPT: coords=0x2357cb30, nverts=3015292, globalToVert=...)

is trying to handle 30M verts which I would not expect to be a problem in terms of memory usage so I think I am hitting an index size issue.

[cwsmith]

What branch/commit were these tests using?

[cwsmith]

Discussion notes:

• This size is concerning: MPT: Can I create issues from the github app on my phone #12 0x000000000048eb5f in PCU_Comm_Pack (to_rank=to_rank@entry=639,
  MPT: data=data@entry=0x2357cb30, size=18446744056901788288)
• n appears to be blowing up here

  core/apf/apfConstruct.cc

  Line 202 in c1d05c1

  PCU_Comm_Pack(to, coords, n*3*sizeof(double));

• Gid is an int...

  core/apf/apfConstruct.cc

  Line 11 in c1d05c1

  typedef int Gid;

• GlobalToVert is an int map ...

  core/apf/apfConvert.h

  Line 33 in c1d05c1

  typedef std::map<int, MeshEntity*> GlobalToVert;

• push changes that start conversion of int to long: fe15d56#diff-8bc0b6c18834c6ae24f81d2049ae3ad8e7859cf8f8f14ad75bf793bf15b40aa2

[KennethEJansen]

MGEN_write3D (c1d05c1)

[KennethEJansen]

Is there a quick way to convert from long to int? As you feared, there is propagation

[ 38%] Building CXX object mds/CMakeFiles/mds.dir/apfBox.cc.o
[ 57%] Built target parma
/home5/kjansen/SCOREC-core/core/mds/apfMDS.cc: In function 'void apf::deriveMdlFromManifold(apf::Mesh2*, bool*, int, int ()[5], apf::GlobalToVert&, std::map<int, apf::MeshEntity>&)':
/home5/kjansen/SCOREC-core/core/mds/apfMDS.cc:1054:55: error: no matching function for call to 'apf::Mesh2::setIntTag(apf::MeshEntity*&, apf::MeshTag*&, const long int*)'
mesh->setIntTag(vit->second, vIDTag, &(vit->first));
^
In file included from /home5/kjansen/SCOREC-core/core/apf/apfMesh2.h:14:0,
from /home5/kjansen/SCOREC-core/core/mds/apfPM.h:14,
from /home5/kjansen/SCOREC-core/core/mds/apfMDS.cc:15:
/home5/kjansen/SCOREC-core/core/apf/apfMesh.h:245:18: note: candidate: virtual void apf::Mesh::setIntTag(apf::MeshEntity*, apf::MeshTag*, const int*)
virtual void setIntTag(MeshEntity* e, MeshTag* tag, int const* data) = 0;
^~~~~~~~~
but I guess I am hoping that with a cast of some sorts on the usage of GlobalToVert, the propagation can be stopped at least for now in routines I am hopefully not using.

[KennethEJansen]

Note I also had to create a PCU_Max_Long which I hopefully replicated correctly from PCU_Max_Int

[KennethEJansen]

I got around the casting issue above but am hitting new issues
Scanning dependencies of target matchedNodeElmReader
[100%] Building CXX object test/CMakeFiles/matchedNodeElmReader.dir/matchedNodeElmReader.cc.o
In file included from /home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc:6:0:
/home5/kjansen/SCOREC-core/core/apf/apfConvertTags.h: In function '{anonymous}::Gid {anonymous}::getMax(const GlobalToVert&)':
/home5/kjansen/SCOREC-core/core/apf/apfConvertTags.h:14:36: error: no matching function for call to 'max({anonymous}::Gid&, const long int&)'
max = std::max(max, it->first);
^
In file included from /nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/bits/stl_tree.h:63:0,
from /nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/map:60,
from /home5/kjansen/SCOREC-core/core/mds/apfMDS.h:33,
from /home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc:3:
/nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/bits/stl_algobase.h:219:5: note: candidate: template constexpr const _Tp& std::max(const _Tp&, const _Tp&)
max(const _Tp& __a, const _Tp& __b)
^~~
/nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/bits/stl_algobase.h:219:5: note: template argument deduction/substitution failed:
In file included from /home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc:6:0:
/home5/kjansen/SCOREC-core/core/apf/apfConvertTags.h:14:36: note: deduced conflicting types for parameter 'const _Tp' ('int' and 'long int')
max = std::max(max, it->first);
^
In file included from /nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/bits/stl_tree.h:63:0,
from /nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/map:60,
from /home5/kjansen/SCOREC-core/core/mds/apfMDS.h:33,
from /home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc:3:
/nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/bits/stl_algobase.h:265:5: note: candidate: template<class _Tp, class _Compare> constexpr const _Tp& std::max(const _Tp&, const _Tp&, _Compare)
max(const _Tp& __a, const _Tp& __b, _Compare __comp)
^~~
/nasa/pkgsrc/sles12/2016Q4/gcc6/include/c++/bits/stl_algobase.h:265:5: note: template argument deduction/substitution failed:
In file included from /home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc:6:0:
/home5/kjansen/SCOREC-core/core/apf/apfConvertTags.h:14:36: note: deduced conflicting types for parameter 'const _Tp' ('int' and 'long int')
max = std::max(max, it->first);
^
/home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc: In function 'int main(int, char**)':
/home5/kjansen/SCOREC-core/core/test/matchedNodeElmReader.cc:825:73: error: cannot convert '{anonymous}::Gid* {aka int*}' to 'const Gid* {aka const long int*}' for argument '2' to 'void apf::construct(apf::Mesh2*, const Gid*, int, int, apf::GlobalToVert&)'
apf::construct(mesh, m.elements, m.localNumElms, m.elementType, outMap);

[jacobmerson]

Where do you define the GID typedef? The error makes it look like its defined in an anonymous namespace.

[cwsmith]

It is/was in apf here:

core/apf/apfConvert.h

Line 32 in fe15d56

typedef long Gid;

I'm hacking at this now.

[KennethEJansen]

Reproducer: It is not asmall as I would like but here is a path to the case on the viz nodes for your you to grab (note it it grabs inputs from one directory above in the run line so safest to just grab the dir above were I am running it).

1. Runs successfully if I roll back to where we were when we tried to make everything with GID long (e.g., here is the hash)
   git checkout c1d05c1

2. here is the case and the successful run

Case DIR /projects/tools/Models/BoeingBump/LES_DNS_Meshing/FPS-MTW-12-30/MGEN4mner_noIDX/mner

 mpirun -np 8 /projects/tools/SCOREC-core/buildLastWorking_mner/test/matchedNodeElmReader ../geom3D.cnn_data ../geom3D.crd ../geom3D.match ../geom3D.class ../geom3D.fathers2D NULL ../geom3DHead.cnn outModel.dmg outMesh/
numVerts 2560663
0 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 0 lastVtx 320082
1 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 320082 lastVtx 640164
2 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 640164 lastVtx 960246
7 readMatches numvtx 2560663 localnumvtx 320089 firstVtx 2240574 lastVtx 2560663
5 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 1600410 lastVtx 1920492
6 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 1920492 lastVtx 2240574
3 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 960246 lastVtx 1280328
4 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 1280328 lastVtx 1600410
isMatched 1
seconds to create mesh 117.841
  - verifying tags: fathers2D
mesh verified in 47.882082 seconds
mesh outMesh/ written in 12.792607 seconds
writeVtuFile into buffers: 5.066701 seconds
writeVtuFile buffers to disk: 1.829939 seconds
vtk files rendered written in 7.314078 seconds

Note though the covert takes 2 minutes (with -g)

3. our current code (run here with output) takes only 20 seconds to crash as follows (in the same dir same data).

mpirun -np 8 /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader ../geom3D.cnn_data ../geom3D.crd ../geom3D.match ../geom3D.class ../geom3D.fathers2D NULL ../geom3DHead.cnn outModel.dmg outMesh/

numVerts 2560663
0 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 0 lastVtx 320082
1 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 320082 lastVtx 640164
2 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 640164 lastVtx 960246
3 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 960246 lastVtx 1280328
4 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 1280328 lastVtx 1600410
5 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 1600410 lastVtx 1920492
6 readMatches numvtx 2560663 localnumvtx 320082 firstVtx 1920492 lastVtx 2240574
7 readMatches numvtx 2560663 localnumvtx 320089 firstVtx 2240574 lastVtx 2560663
isMatched 1
[viz002:25725] *** Process received signal ***
[viz002:25725] Signal: Segmentation fault (11)
[viz002:25725] Signal code: Address not mapped (1)
[viz002:25725] Failing at address: (nil)
[viz002:25725] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f70e3098890]
[viz002:25725] [ 1] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x4ab948]
[viz002:25725] [ 2] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x4abb2d]
[viz002:25725] [ 3] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x4abc66]
[viz002:25725] [ 4] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x4ac630]
[viz002:25725] [ 5] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(mds_create_entity+0xe1)[0x4accbc]
[viz002:25725] [ 6] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(mds_apf_create_entity+0x49)[0x4adaca]
[viz002:25725] [ 7] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf7MeshMDS13createEntity_EiPNS_11ModelEntityEPPNS_10MeshEntityE+0x13f)[0x49b579]
[viz002:25725] [ 8] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf5Mesh212createEntityEiPNS_11ModelEntityEPPNS_10MeshEntityE+0x44)[0x4d04bc]
[viz002:25725] [ 9] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf10makeOrFindEPNS_5Mesh2EPNS_11ModelEntityEiPPNS_10MeshEntityEPNS_13BuildCallbackEPb+0x6a)[0x4ceda0]
[viz002:25725] [10] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf14ElementBuilder5applyEiPPNS_10MeshEntityE+0x43)[0x4d0559]
[viz002:25725] [11] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf13ElementVertOp3runEiPPNS_10MeshEntityE+0x44)[0x4c8a2a]
[viz002:25725] [12] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x4c84c5]
[viz002:25725] [13] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf13ElementVertOp7runDownEiPPNS_10MeshEntityES3_+0x3d)[0x4c89e3]
[viz002:25725] [14] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf13ElementVertOp3runEiPPNS_10MeshEntityE+0x2a)[0x4c8a10]
[viz002:25725] [15] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x4c8658]
[viz002:25725] [16] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf13ElementVertOp7runDownEiPPNS_10MeshEntityES3_+0x3d)[0x4c89e3]
[viz002:25725] [17] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf13ElementVertOp3runEiPPNS_10MeshEntityE+0x2a)[0x4c8a10]
[viz002:25725] [18] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf12buildElementEPNS_5Mesh2EPNS_11ModelEntityEiPPNS_10MeshEntityEPNS_13BuildCallbackE+0x48)[0x4cee21]
[viz002:25725] [19] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x501d44]
[viz002:25725] [20] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(_ZN3apf9constructEPNS_5Mesh2EPKliiRSt3mapIlPNS_10MeshEntityESt4lessIlESaISt4pairIS2_S6_EEE+0x54)[0x5024e6]
[viz002:25725] [21] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader(main+0x1ea)[0x48eb2c]
[viz002:25725] [22] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f70e2cffb45]
[viz002:25725] [23] /projects/tools/SCOREC-core/buildDbg/test/matchedNodeElmReader[0x48b488]
[viz002:25725] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 2 with PID 25725 on node viz002 exited on signal 11 (Segmentation fault).

edit: formatting

[KennethEJansen]

I decided to try serial. Long-mod code makes it through construct in serial but then segfaults on the line 138 in mds_apf.c on return gmi_find(m-> user_model, ....
I made a debug executable rolled back to before we moved int to long.

what totalview showed me for the mesh (m) was pretty messed up so its note REALLY getting through construct cleanly. I have a new run stopping just after constuct to check that but it is probably time to build with memory sanitizer.

OK it got to my break just before delete [] m.elements on line 826 of matchedNodeElemReader.

m.elements are junk as the first entry is 140007343915089 according to totalview

I will move my search into where this is assigned to see what is going wrong.