TEST: add buffers for onesided tests

[wfaderhold21]

What

Adds extra src/dst/work buffers for onesided algorithms in the MPI test.

Why ?

These are necessary to ensure RDMA writes are not performed on the next test before all processes have completed the initial test.

[swx-jenkins3]

Can one of the admins verify this patch?

[samnordmann]

Thanks @wfaderhold21
We briefly discussed this PR during our WG. Can you please explain what precisely this patch aims to fix and why we need both a barrier and doubling the buffers?

[wfaderhold21]

Thanks @wfaderhold21 We briefly discussed this PR during our WG. Can you please explain what precisely this patch aims to fix and why we need both a barrier and doubling the buffers?

Sure.

The aim of this patch is to fix a potential issue when running the MPI test when multiple transports (i.e., RC + XPMEM) are being used. This could allow for one process (e.g., rank 0) to finish the collective and move to the next collective before another process (e.g., rank 1) has finished the check on its destination buffer.

The use of multiple buffers was to allow for processes to alternate between the buffers and prevent overwriting of a destination buffer because onesided operations are being used. However, this could be simply reduced to a barrier before collective start, which will prevent a process from entering the next collective while another process is checking the result of the previous collective.

[wfaderhold21]

@janjust are we including this for 1.5?

[janjust]

yes - but we have several critical PRs to go through before, we will do one rebase at a time until we can merge them all.

[Sergei-Lebedev]

ok to test