UCX Integration

[supunkamburugamuve]

Integrate UCX with Cylon

[benjha]

Hi Cylon team,

I am curious about this topic. Currently, OpenMPI can use UCX as its transport layer. It works quite good under infiniband and supports "fancy things" like cuda aware MPI, etc.

What is the rationale of this integration when it is possible to use Cylon with UCX via OpenMPI now?

Thanks!

[chathurawidanage]

Hi Benjamin, With UCX, we get the flexibility to spawn processes anywhere freely without having to rely on mpirun or OpenMPI to start the processes. This makes running Cylon on Kubernetes and scaling(could be dynamic as well) straightforward when coupled with a master or a driver program(scheduler). Other than that, with MPI we have to stick to the worker = process model(unless we are fine with managing a large complex codebase ☺). With UCX, we can easily change this to a worker = thread model, if required. All these things are future work, but UCX will be the base for them. Regards, Chathura

…

On Thu, Jun 24, 2021 at 11:07 PM Benjamín Hernández < ***@***.***> wrote: Hi Cylon team, I am curious about this topic. Currently, OpenMPI can use UCX as its transport layer. It works quite good under infiniband and supports "fancy things" like cuda aware MPI, etc. What is the rationale of this integration when it is possible to use Cylon with UCX via OpenMPI now? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#281 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXZKCMUEP4FPUZRSUMVZU3TUPXG3ANCNFSM4V7WTJXA> .

[chathurawidanage]

Also, if we need to do distributed data processing inside an existing framework(ie. Storm etc.) that has a set of pre spawned processes, with UCX we could start a Cylon network between those processes to offload data processing. So it helps us to move Cylon towards its original goal "Data Engineering Everywhere!".

[benjha]

Thanks @chathurawidanage for the explanation. Now I understand better the context, and yeah, it is much better to have a transport based on UCX instead of TCP as most of the analytics tools do.