Will the future release supports DSM (Distributed Shared Memory) ?

[wyoung1]

Hi, I notice that one of the core feature: "Immutable zero-cost in-memory data sharing via memory mapping " only supports in IPC client with local Vineyard instances. Will the future release supports DSM (Distributed Shared Memory) ?

[dashanji]

Hi @wyoung1, currently Vineyard supports both the IPC client and the RPC client. For more info, please refer to the doc https://v6d.io/notes/architecture.html#client-side

BTW, would you please elaborate a case why we need the DSM?

[wyoung1]

Hi @dashanji , I suppose, the DSM may supports zero-copy data sharing while we use a remote object, since the current migration process takes a considerable time.

[dashanji]

Hi @dashanji , I suppose, the DSM may supports zero-copy data sharing while we use a remote object, since the current migration process takes a considerable time.

Cool! It sounds great that vineyard can support zero-copy data sharing on distributed machines. Actually, I have no idea about DSM. Do we need to run the Vineyard on a special hardware or a shared memory platform? Could you please share some background here? Thanks!

[wyoung1]

haha, I have little idea about this too and I'm currently doing some research, maybe RDMA or CXL will help.

Hi @dashanji , I suppose, the DSM may supports zero-copy data sharing while we use a remote object, since the current migration process takes a considerable time.

Cool! It sounds great that vineyard can support zero-copy data sharing on distributed machines. Actually, I have no idea about DSM. Do we need to run the Vineyard on a special hardware or a shared memory platform? Could you please share some background here? Thanks!

[wyoung1]

maybe this would help : https://github.com/etascale/argodsm/tree/master

[dashanji]

Thanks！

[sighingnow]

Hi @wyoung1,

Thanks for raising the question! Compared with distributed shared memory, vineyard is more like a distributed object store.

We do have taken distributed shared memory (or far memroy, like AIFM) into consideration, but we finally chose "object store for anything" as the user interfaces, for the following reasons:

• to maximize the performance, the DSM should be application-integrated
• current data analytical workloads (e.g., Spark, distributed training) already have data partition or chunking logic, i.e., each worker manages parts of the global data. Vineyard built the same abstraction for distributed objects, i.e., Collection<T> in vineyard. E.g., a distributed dataframe is Collection<DataFrame>.

However, that doesn't mean vineyard cannot support DSM. You can imagine vineyard is the local memory allocator deployed in each machine, and built a DSM layer on top of vineyard's shared memory manager.

Actually, we have external users who are currently building a CXL-based distributed memory systems on top of vineyard, where vineyard serves as the local memory manager and metadata service.

[cTatu]

Hello, could you give more information on the users' projects of building a CXL-based distributed memory systems on top of vineyard?
We are interested in a distributed object store with the ability to use rdma/infiniband to transfer objects between nodes.

[dashanji]

Hi @cTatu. Thanks for your attention.

We are currently in the process of adding RDMA capability to Vineyard. Stay tuned for updates!

[sighingnow]

@wyoung1 If there are any potential use case or scenario that vineyard can help to serve as a distributed shared memory manager, please let we know!

Thanks!

[wyoung1]

Hi @wyoung1,

Thanks for raising the question! Compared with distributed shared memory, vineyard is more like a distributed object store.

We do have taken distributed shared memory (or far memroy, like AIFM) into consideration, but we finally chose "object store for anything" as the user interfaces, for the following reasons:

• to maximize the performance, the DSM should be application-integrated
• current data analytical workloads (e.g., Spark, distributed training) already have data partition or chunking logic, i.e., each worker manages parts of the global data. Vineyard built the same abstraction for distributed objects, i.e., Collection<T> in vineyard. E.g., a distributed dataframe is Collection<DataFrame>.

However, that doesn't mean vineyard cannot support DSM. You can imagine vineyard is the local memory allocator deployed in each machine, and built a DSM layer on top of vineyard's shared memory manager.

Actually, we have external users who are currently building a CXL-based distributed memory systems on top of vineyard, where vineyard serves as the local memory manager and metadata service.

Ok, thanks! A great case, I'll have a try.