Node Level Scaling on KNL (on Cori)

On this page, we track the state of node-level scaling on KNL (on Cori).

Current state

• Pretty good node-level scaling
• Strong improvement compared to mid-January
• Unexplained drops for some number of threads
• Ceveat: Old results computed single timestep, new results are for 30 timesteps
• Missing data in the graphs is due to not enough walltime in the job configuration The following graph shows the runtime for different numbers of hpx threads ("-hpx:threads").

• Level 9 doesn't fit into the MCDRAM

This corresponds to the parallel efficiency displayed in the next graph.

Console output to individual experiments can be found here.

Reproduce

srun numactl -m 1 ./knl-build/octotiger-Release/octotiger         \
         -Disableoutput                                           \
         -Problem=dwd                                             \
         -Max_level=${level}                                      \
         -Xscale=4.0                                              \
         -Eos=wd                                                  \
         -Angcon=1                                                \
         -Stopstep=0.01                                           \
         --hpx:threads=${threads}                                 \
         -Restart=restart${level}.chk                             \
         --hpx:ini=hpx.stacks.small_size=0xC0000                  \
         -Ihpx.stacks.use_guard_pages=0                           \
         --hpx:print-bind                                         \
         --hpx:print-counter /threads{locality#*/total}/idle-rate \
         > results/${name}_N${SLURM_NNODES}_t${threads}_l${level}_m1 2>&1

Parameters for older graphs (moving star problem):

srun numactl -m 1 ./knl-build/octotiger-Release/octotiger     \
     -Disableoutput                                           \
     -Problem=moving_star                                     \
     -Max_level=${level}                                      \
     -Stopstep=0 --hpx:threads=${threads}                     \
     --hpx:ini=hpx.stacks.small_size=0xC00000                 \
     -Ihpx.stacks.use_guard_pages=0                           \
     --hpx:print-bind                                         \
     --hpx:print-counter /threads{locality#*/total}/idle-rate \
     > results/${name}_N${SLURM_NNODES}_t${threads}_l${level}_m1 2>&1

• These parameters will result in an out-of-memory error for level 5 with MCDRAM and level 6 with DRAM