TEST/MPI: Added MPI+CUDA example.

[rakhmets]

What?

Added MPI+CUDA example.

[Akshay-Venkatesh]

@rakhmets Overall the tests stress multi-GPU support in a good way.

I see the following not being tested:

1. The case where one thread has device context bound to it and it allocates device memory. Later another thread issues the MPI operations with the allocated memory.
2. Also use cudaSetDevice/cudaDeviceReset runtime API Instead of explicitly using ctxRetain/Release. This is more commonly the API exercised by high-level applications so it would be good to ensure that no cases break from just testing driver API.

The above two cases are supported by multi-GPU support, right? If so, will they be tested in a separate tests?

[Akshay-Venkatesh]

@rakhmets Minor comment -- By this logic, if the MPI job were to be launched on the same node, both ranks would end up initializing cu_dev to 0 by the end of the above loop. If there are multiple GPUs on the node, I think it would be good to check where the ranks manage different GPUs. As an example, osu benchmarks use LOCAL_RANK env for this reason.

[rakhmets]

Since the test is only for two processes, I fixed this by calling cuDeviceGet(&cu_dev, 1) for the 2nd rank.

[rakhmets]

@rakhmets Overall the tests stress multi-GPU support in a good way.

I see the following not being tested:

1. The case where one thread has device context bound to it and it allocates device memory. Later another thread issues the MPI operations with the allocated memory.
2. Also use cudaSetDevice/cudaDeviceReset runtime API Instead of explicitly using ctxRetain/Release. This is more commonly the API exercised by high-level applications so it would be good to ensure that no cases break from just testing driver API.

The above two cases are supported by multi-GPU support, right? If so, will they be tested in a separate tests?

1. test_alloc_prim_send_no does pretty much the same thing (from the perspective of the feature implementation). There is an active (retained) primary device context, but it is not bound to the thread at the moment of MPI send/recv.
2. I think it would be better to have separate test for CUDA Runtime API only. Because some scenarios (e.g. create user context) are only valid for Driver API.

I will add a separate test using CUDA Runtime API (probably in another PR). And I will include the test case described in the first bullet to this new test.