Rapids integration

[supunkamburugamuve]

We discussed about two approaches for integrating with Rapids

1. Implement our own alltoall for Rapids and implement the distributed operations
2. Implement Cylon Dask plugin

[ahmet-uyar]

GPU support seems to be added to Spark 3.
https://developer.nvidia.com/blog/accelerating-apache-spark-3-0-with-gpus-and-rapids/
https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/apache-spark-3/

"NVIDIA has also created a new Spark shuffle implementation that optimizes the data transfer between Spark processes. This shuffle implementation is built upon GPU-accelerated communication libraries, including UCX, RDMA, and NCCL."

[ahmet-uyar]

They are also using CuDF at GPUs:
https://nvidia.github.io/spark-rapids/
https://github.com/nvidia/spark-rapids

[ahmet-uyar]

they seem to be implemented shuffling with scala at spark-rapids:
https://github.com/NVIDIA/spark-rapids/tree/branch-0.4/shuffle-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/ucx

[supunkamburugamuve]

The BLINK paper is about heterogeneous environments where there can be slow and fast connections between different GPUs. So it doesn't have a connection with the task-based execution.

https://proceedings.mlsys.org/paper/2020/file/43ec517d68b6edd3015b3edc9a11367b-Paper.pdf

[ahmet-uyar]

I think in our case, if we can do something like RAPIDS-accelerated Spark shuffles, that would be sufficient. If communicating GPUs are on the same node and NVLink is avilable, they use NVLink to communicate. If communicating GPUs are on separate nodes, then they use RDMA. In both cases, they avoid going through PCI-e bus and CPUs.

Blink seems to be a little lower level. They also say that Blink provides NCCL-compatible API. It does not requires user program modifications and only relies on preloading (LD PRELOAD).

[supunkamburugamuve]

Attaching the file we drew in the discussion