PCU: Add nompi stub

[bobpaw]

Add nompi stub

Add a stub implementation of MPI in pcu/pcu_pnompi.c.
Remove direct calls to MPI functions.

This adds a SCOREC_NO_MPI build option for users running on single-process, single-machine setups that do not want to compile with MPI. Message passing is simulated in memory.

[bobpaw]

@cwsmith, build is failing for this PR. I think we should add PCU_Init and PCU_Finalize facades that wrap the MPI calls and do a PCU_Comm_Init() and PCU_Comm_Free() to remove the direct dependencies. There are a lot of executables that would need to be changed and this would become a large PR. Alternatively, we could define a replacement MPI_Init and MPI_Finalize in pcu_pnompi and link to those. Or, we could do a hybrid approach and encourage people to use PCU_Init in the future.

[jacobmerson]

@flagdanger and I are working on a very large update of PCU that should make this all easier by getting rid of the singleton. Pull request for that coming within the next week.

[jacobmerson]

@bobpaw sorting out a few things to make the pull request ready to merge, but #388 the draft pull request for replacing the singleton in PCU.

Although we replaced the interface used in the test cases, this pull request is designed to work with no modification to existing codes by using a global PCU state when the old style interface is used with PCU_Comm_Init or it can use local state.

[bobpaw]

@jacobmerson, we will plan to use your PR for the future of core, but the sponsor wants this functionality soon, so we will use this branch until yours is finished.

[cwsmith]

@bobpaw Does PCU_Comm_Init() check if MPI was initialized?

[bobpaw]

@bobpaw Does PCU_Comm_Init() check if MPI was initialized?

No, at the moment it only checks if PCU has been initialized using a global state variable. We could check with MPI_Initialized.

[onkarsahni]

@cwsmith Note Aiden is only doing this when SCOREC_NO_MPI is defined to be true, so there is no actual MPI (e.g., see pcu/pcu_pnompi.c). For now, he is not looking into PCU_init, that will be done after the other PR #388.

[cwsmith]

@bobpaw OK. I'm wondering if it will make sense to make the MPI check in that call, and if it fails call MPI_Init. But... handling MPI_Finalize could be a bit more tricky; other libraries may still be running and using MPI beyond the lifetime of PCU. I'd prefer symmetry in the interface so your original suggestion may be best.

[bobpaw]

@cwsmith I have made the changes you and Jacob requested. Can you review when you get the chance?

[cwsmith]

@bobpaw Here is a comment that discusses running the delta wing regression test I mentioned earlier:
#433 (comment)

[bobpaw]

@bobpaw Here is a comment that discusses running the delta wing regression test I mentioned earlier: #433 (comment)

Do I need to worry about about timing? Or should I just compare the quality file?

[cwsmith]

We should check timing and quality.

[bobpaw]

We should check timing and quality.

@cwsmith, adapt completed in 5761.497560 seconds. How do I measure the resulting length_proc.txt and quality_proc.txt files? I see there's a histogram utility but it wants min, max, nbins.

[cwsmith]

@bobpaw Hmmm. I'm guessing that was a serial run? If so, I only have prior results for a 32 processes (see #433 (comment)). Regarding quality, if the final entity counts are similar we should be OK. We could also render the adapted mesh to sanity check.

[bobpaw]

No, this was a 32proc run on wilbur.

[bobpaw]

number of tet 5934317 hex 0 prism 0 pyramid 0
mesh entity counts: v 1067369 e 7026323 f 11893272 r 5934317

[bobpaw]

But it was a debug build without optimization so let me rerun with release optimizations.

[cwsmith]

For the record, wilbur and piglet have the same hardware.

[bobpaw]

Wow, I'm surprised how much optimizations improved the runtime:

MeshAdapt: mesh adapted in 308.636508 seconds
number of tet 5934317 hex 0 prism 0 pyramid 0
mesh entity counts: v 1067369 e 7026323 f 11893272 r 5934317

[cwsmith]

That time looks good. Can you please paste/post your cmake config command and the output log?

[bobpaw]

config/deltaWing.sh:

#!/bin/sh

build_id=deltaWing
if [ $# -gt 0 ]; then
  build_id=$1
fi

cmake -S . -B build/$build_id \
  -GNinja \
  -DCMAKE_INSTALL_PREFIX="$(realpath install)/$build_id" \
  -DCMAKE_BUILD_TYPE="Release" \
  -DCMAKE_C_COMPILER="mpicc" \
  -DCMAKE_CXX_COMPILER="mpicxx" \
  -DSCOREC_CXX_SYMBOLS=OFF \
  -DSCOREC_CXX_OPTIMIZE=ON \
  -DSCOREC_CXX_WARNINGS=ON \
  -DBUILD_SHARED_LIBS=ON \
  -DBUILD_EXES=ON \
  -DIS_TESTING=ON \
  -DENABLE_ZOLTAN=ON \
  -DENABLE_SIMMETRIX=ON -DSKIP_SIMMETRIX_VERSION_CHECK=ON \
  -DENABLE_FIELDSIM=OFF -DSIM_MPI=mpich4.1.1 -DSIM_ACIS=ON -DSIM_PARASOLID=ON \
  -DPCU_COMPRESS=ON

run_adapt.log

model /lore/smithc11/projects/deltaWingAdapt/pumiAdapt/deltaWing.dmg loaded in 0.000793 seconds
mesh p32/ loaded in 0.062518 seconds
number of tet 581196 hex 0 prism 0 pyramid 0
mesh entity counts: v 104305 e 691152 f 1168044 r 581196
  - verifying tags: mach_ver, original_metric_ver, target_metric_ver
mesh verified in 0.148043 seconds
writeVtuFile into buffers: 0.030325 seconds
writeVtuFile buffers to disk: 0.005144 seconds
vtk files anisoDelta_before written in 0.038241 seconds

MeshAdapt: ma::configure:  Based on requested sizefield, MeshAdapt requires at least 3 iterations.
           Setting the number of iteration to 4!

MeshAdapt: version 2.0 - dev !
planned Zoltan balance to target imbalance 1.100000 in 0.220910 seconds
Zoltan balanced to 1.100000 in 0.524734 seconds

MeshAdapt: iteration 0

MeshAdapt: coarsened 53 edges in 0.272583 seconds

MeshAdapt: refined 611639 edges in 3.049696 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 8683

MeshAdapt: --fixLargeAngles       in 0.430228 seconds: #bad elements 2235

MeshAdapt: --fixShortEdgeElements in 1.284403 seconds: #bad elements 818

MeshAdapt: --percent change in number of bad elements 0.905793

MeshAdapt: --iter 1 of shape correction loop: #bad elements 818

MeshAdapt: --fixLargeAngles       in 0.104712 seconds: #bad elements 288

MeshAdapt: --fixShortEdgeElements in 0.358577 seconds: #bad elements 143

MeshAdapt: --percent change in number of bad elements 0.825183

MeshAdapt: --iter 2 of shape correction loop: #bad elements 143

MeshAdapt: --fixLargeAngles       in 0.043554 seconds: #bad elements 82

MeshAdapt: --fixShortEdgeElements in 0.230717 seconds: #bad elements 58

MeshAdapt: --percent change in number of bad elements 0.594406

MeshAdapt: --iter 3 of shape correction loop: #bad elements 58

MeshAdapt: --fixLargeAngles       in 0.035118 seconds: #bad elements 56

MeshAdapt: --fixShortEdgeElements in 0.198542 seconds: #bad elements 53

MeshAdapt: --percent change in number of bad elements 0.086207

MeshAdapt: --iter 4 of shape correction loop: #bad elements 53

MeshAdapt: --fixLargeAngles       in 0.034678 seconds: #bad elements 52

MeshAdapt: --fixShortEdgeElements in 0.147253 seconds: #bad elements 49

MeshAdapt: --percent change in number of bad elements 0.075472

MeshAdapt: --iter 5 of shape correction loop: #bad elements 49

MeshAdapt: --fixLargeAngles       in 0.032862 seconds: #bad elements 52

MeshAdapt: --fixShortEdgeElements in 0.145992 seconds: #bad elements 49

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 8683 to 49 in 31.149691 seconds

MeshAdapt: iteration 1

MeshAdapt: coarsened 10588 edges in 2.884405 seconds

MeshAdapt: refined 508001 edges in 4.174828 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 21415

MeshAdapt: --fixLargeAngles       in 0.773972 seconds: #bad elements 2934

MeshAdapt: --fixShortEdgeElements in 0.928731 seconds: #bad elements 556

MeshAdapt: --percent change in number of bad elements 0.974037

MeshAdapt: --iter 1 of shape correction loop: #bad elements 556

MeshAdapt: --fixLargeAngles       in 0.114474 seconds: #bad elements 150

MeshAdapt: --fixShortEdgeElements in 0.323925 seconds: #bad elements 71

MeshAdapt: --percent change in number of bad elements 0.872302

MeshAdapt: --iter 2 of shape correction loop: #bad elements 71

MeshAdapt: --fixLargeAngles       in 0.050748 seconds: #bad elements 40

MeshAdapt: --fixShortEdgeElements in 0.133650 seconds: #bad elements 32

MeshAdapt: --percent change in number of bad elements 0.549296

MeshAdapt: --iter 3 of shape correction loop: #bad elements 32

MeshAdapt: --fixLargeAngles       in 0.050780 seconds: #bad elements 28

MeshAdapt: --fixShortEdgeElements in 0.070426 seconds: #bad elements 28

MeshAdapt: --percent change in number of bad elements 0.125000

MeshAdapt: --iter 4 of shape correction loop: #bad elements 28

MeshAdapt: --fixLargeAngles       in 0.049759 seconds: #bad elements 32

MeshAdapt: --fixShortEdgeElements in 0.070114 seconds: #bad elements 28

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 21415 to 28 in 38.095028 seconds

MeshAdapt: iteration 2

MeshAdapt: coarsened 145151 edges in 7.693994 seconds

MeshAdapt: refined 31824 edges in 1.503134 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 1294

MeshAdapt: --fixLargeAngles       in 0.195557 seconds: #bad elements 226

MeshAdapt: --fixShortEdgeElements in 0.222646 seconds: #bad elements 32

MeshAdapt: --percent change in number of bad elements 0.975270

MeshAdapt: --iter 1 of shape correction loop: #bad elements 32

MeshAdapt: --fixLargeAngles       in 0.045653 seconds: #bad elements 17

MeshAdapt: --fixShortEdgeElements in 0.112203 seconds: #bad elements 12

MeshAdapt: --percent change in number of bad elements 0.625000

MeshAdapt: --iter 2 of shape correction loop: #bad elements 12

MeshAdapt: --fixLargeAngles       in 0.045051 seconds: #bad elements 11

MeshAdapt: --fixShortEdgeElements in 0.066190 seconds: #bad elements 11

MeshAdapt: --percent change in number of bad elements 0.083333

MeshAdapt: --iter 3 of shape correction loop: #bad elements 11

MeshAdapt: --fixLargeAngles       in 0.047230 seconds: #bad elements 11

MeshAdapt: --fixShortEdgeElements in 0.057770 seconds: #bad elements 10

MeshAdapt: --percent change in number of bad elements 0.090909

MeshAdapt: --iter 4 of shape correction loop: #bad elements 10

MeshAdapt: --fixLargeAngles       in 0.044678 seconds: #bad elements 10

MeshAdapt: --fixShortEdgeElements in 0.057195 seconds: #bad elements 10

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 1294 to 10 in 31.419985 seconds

MeshAdapt: iteration 3

MeshAdapt: coarsened 29195 edges in 3.057790 seconds

MeshAdapt: refined 4222 edges in 0.570225 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 158

MeshAdapt: --fixLargeAngles       in 0.100734 seconds: #bad elements 56

MeshAdapt: --fixShortEdgeElements in 0.192765 seconds: #bad elements 16

MeshAdapt: --percent change in number of bad elements 0.898734

MeshAdapt: --iter 1 of shape correction loop: #bad elements 16

MeshAdapt: --fixLargeAngles       in 0.044524 seconds: #bad elements 13

MeshAdapt: --fixShortEdgeElements in 0.056625 seconds: #bad elements 11

MeshAdapt: --percent change in number of bad elements 0.312500

MeshAdapt: --iter 2 of shape correction loop: #bad elements 11

MeshAdapt: --fixLargeAngles       in 0.046506 seconds: #bad elements 10

MeshAdapt: --fixShortEdgeElements in 0.056879 seconds: #bad elements 10

MeshAdapt: --percent change in number of bad elements 0.090909

MeshAdapt: --iter 3 of shape correction loop: #bad elements 10

MeshAdapt: --fixLargeAngles       in 0.044946 seconds: #bad elements 10

MeshAdapt: --fixShortEdgeElements in 0.055969 seconds: #bad elements 10

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 158 to 10 in 23.725210 seconds

MeshAdapt: iteration 4

MeshAdapt: coarsened 4361 edges in 1.933079 seconds

MeshAdapt: refined 667 edges in 0.528031 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 55

MeshAdapt: --fixLargeAngles       in 0.099377 seconds: #bad elements 29

MeshAdapt: --fixShortEdgeElements in 0.114539 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.836364

MeshAdapt: --iter 1 of shape correction loop: #bad elements 9

MeshAdapt: --fixLargeAngles       in 0.043805 seconds: #bad elements 9

MeshAdapt: --fixShortEdgeElements in 0.060760 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 55 to 9 in 11.790645 seconds

MeshAdapt: iteration 5

MeshAdapt: coarsened 665 edges in 1.713284 seconds

MeshAdapt: refined 164 edges in 0.503225 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 35

MeshAdapt: --fixLargeAngles       in 0.094527 seconds: #bad elements 28

MeshAdapt: --fixShortEdgeElements in 0.109800 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.742857

MeshAdapt: --iter 1 of shape correction loop: #bad elements 9

MeshAdapt: --fixLargeAngles       in 0.045398 seconds: #bad elements 9

MeshAdapt: --fixShortEdgeElements in 0.060482 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 35 to 9 in 11.793450 seconds

MeshAdapt: iteration 6

MeshAdapt: coarsened 162 edges in 1.659875 seconds

MeshAdapt: refined 74 edges in 0.500956 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 30

MeshAdapt: --fixLargeAngles       in 0.093688 seconds: #bad elements 28

MeshAdapt: --fixShortEdgeElements in 0.109019 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.700000

MeshAdapt: --iter 1 of shape correction loop: #bad elements 9

MeshAdapt: --fixLargeAngles       in 0.043221 seconds: #bad elements 9

MeshAdapt: --fixShortEdgeElements in 0.059726 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 30 to 9 in 11.783985 seconds

MeshAdapt: iteration 7

MeshAdapt: coarsened 62 edges in 1.573582 seconds

MeshAdapt: refined 35 edges in 0.505162 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 29

MeshAdapt: --fixLargeAngles       in 0.096616 seconds: #bad elements 28

MeshAdapt: --fixShortEdgeElements in 0.110339 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.689655

MeshAdapt: --iter 1 of shape correction loop: #bad elements 9

MeshAdapt: --fixLargeAngles       in 0.043220 seconds: #bad elements 9

MeshAdapt: --fixShortEdgeElements in 0.059952 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 29 to 9 in 11.728326 seconds

MeshAdapt: iteration 8

MeshAdapt: coarsened 26 edges in 1.632900 seconds

MeshAdapt: refined 34 edges in 0.506708 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 28

MeshAdapt: --fixLargeAngles       in 0.044710 seconds: #bad elements 28

MeshAdapt: --fixShortEdgeElements in 0.110796 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.678571

MeshAdapt: --iter 1 of shape correction loop: #bad elements 9

MeshAdapt: --fixLargeAngles       in 0.043211 seconds: #bad elements 9

MeshAdapt: --fixShortEdgeElements in 0.059862 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 28 to 9 in 11.647666 seconds

MeshAdapt: iteration 9

MeshAdapt: coarsened 19 edges in 1.560987 seconds

MeshAdapt: refined 24 edges in 0.507650 seconds

MeshAdapt: --iter 0 of shape correction loop: #bad elements 28

MeshAdapt: --fixLargeAngles       in 0.045709 seconds: #bad elements 28

MeshAdapt: --fixShortEdgeElements in 0.110187 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.678571

MeshAdapt: --iter 1 of shape correction loop: #bad elements 9

MeshAdapt: --fixLargeAngles       in 0.044045 seconds: #bad elements 9

MeshAdapt: --fixShortEdgeElements in 0.059999 seconds: #bad elements 9

MeshAdapt: --percent change in number of bad elements 0.000000

MeshAdapt: bad shapes down from 28 to 9 in 11.919131 seconds

MeshAdapt: Maximum (metric) edge length in the mesh is 1.709849

MeshAdapt: 0th additional refine-snap call

MeshAdapt: refined 19 edges in 0.540297 seconds

MeshAdapt: Maximum (metric) edge length in the mesh is 1.499999

MeshAdapt: worst element quality is 1.697484e-02

MeshAdapt: worst element quality is 1.697484e-02
planned Zoltan balance to target imbalance 1.100000 in 2.036767 seconds
Zoltan balanced to 1.100000 in 7.346896 seconds

MeshAdapt: element imbalance 9% of average

MeshAdapt: mesh adapted in 308.030375 seconds
number of tet 5934317 hex 0 prism 0 pyramid 0
mesh entity counts: v 1067369 e 7026323 f 11893272 r 5934317
  - verifying tags: mach_ver, original_metric_ver, target_metric_ver
mesh verified in 1.533155 seconds

MeshAdapt: ma::configure:  Based on requested sizefield, MeshAdapt requires at least 1 iterations.
           Setting the number of iteration to 2!
writeVtuFile into buffers: 0.436119 seconds
writeVtuFile buffers to disk: 0.027537 seconds
vtk files anisoDelta_after written in 0.478725 seconds

[cwsmith]

The timing and counts look good. Here are the final counts from my old run on piglet:

MeshAdapt: mesh adapted in 450.370630 seconds
number of tet 5934317 hex 0 prism 0 pyramid 0
mesh entity counts: v 1067369 e 7026323 f 11893272 r 5934317
  - verifying tags: mach_ver, original_metric_ver, target_metric_ver
mesh verified in 1.516892 seconds

[bobpaw]

Ok, great! Anything else I need to do before PR is approved?

[jacobmerson]

All set from my end.

[cwsmith]

Looks good. Thank you. I left a couple comments below.

[bobpaw]

Do you think I should rename pcu::PCU_Init to pcu::Init? same suggestion for finalize. The C api I would keep the same. I think it's still namespaced and sorter is nicer.

[cwsmith]

Do you think I should rename pcu::PCU_Init to pcu::Init? same suggestion for finalize. The C api I would keep the same. I think it's still namespaced and sorter is nicer.

Either way is fine with me.

Another change request: we need to increase the pumi version number to 4.0.0 in CMakeLists.txt (

core/CMakeLists.txt

Line 9 in 17ad033

project(SCOREC VERSION 3.0.3 LANGUAGES CXX C)

) as we have changed the api (https://semver.org/).

[bobpaw]

Do you think I should rename pcu::PCU_Init to pcu::Init? same suggestion for finalize. The C api I would keep the same. I think it's still namespaced and sorter is nicer.

Either way is fine with me.

Another change request: we need to increase the pumi version number to 4.0.0 in CMakeLists.txt (

core/CMakeLists.txt

Line 9 in 17ad033

project(SCOREC VERSION 3.0.3 LANGUAGES CXX C)

) as we have changed the api (https://semver.org/).

Is there anything we want to bundle with this major release?

[cwsmith]

Is there anything we want to bundle with this major release?

Not that I can think of.

[cwsmith]

/runtests

[cwsmith]

@bobpaw Once the pumi version is increased we should be good to go. Thanks for all the work on this.

[github-actions]

Build Log
Simmetrix Test Result: failure
Simmetrix + CGNS Test Result: skipped

[cwsmith]

@bobpaw The CI tests caught what I'm guessing is a missing header?

 /space/smithc11/githubSelfHostedRunner/actions-runner/_work/core/core/core_425/test/sim_countBL.cc: In function 'int main(int, char**)':
/space/smithc11/githubSelfHostedRunner/actions-runner/_work/core/core/core_425/test/sim_countBL.cc:25:3: error: 'pcu' has not been declared
   25 |   pcu::Init(&argc, &argv);
      |   ^~~
/space/smithc11/githubSelfHostedRunner/actions-runner/_work/core/core/core_425/test/sim_countBL.cc:110:3: error: 'pcu' has not been declared
  110 |   pcu::Finalize();
      |   ^~~