Can v6d support graph data's incremental update?

[songqing]

Describe your problem

Now, v6d seems only support graph data's full update, in other words, loading the graph data once and then serving. It also supports adding new labels, but for the existed label, it seems the data can not be changed.
Loading the full data every time may be time-costed. If we have a full dataset and other small incremental dataset, supporting incremental update may be more efficient.

---

If it is a feature request, please provides a clear and concise description of what you want to happen:

What is the problem:

v6d's data incremental update for graph

The behaviour that you expect to work:

We can load the full data at first, and then load some small incremental updated data for the existed label.

[sighingnow]

Thanks for posting. We are planning to support graph's data incremental update in two folds:

• Add bulk data to existing vertex/edge labels

  Tracked by Support adding the same label edges with different labels in src and dst #1295 and is hopefully finished by the end of this August.

• Continuously adding vertices/edges in a streaming fashion to the existing graph

  We are working on a project called GART which maintains a mutable graph on top of vineyard with snapshot isolation and supports feeding streaming graph updates, which will be open-sourced soon. Stay tuned!

[songqing]

OK, thank you, looking forward to the streaming graph computation project GART. It seems there is little open-source implementation about it.

And for the bulk update of graph data, can you simply introduce the implementation idea?

[sighingnow]

OK, thank you, looking forward to the streaming graph computation project GART. It seems there is little open-source implementation about it.

Thanks for being interested. See also https://mp.weixin.qq.com/s/AyQGbGOfeptNxh_RayPxvA for a short announcement about the GART paper.

And for the bulk update of graph data, can you simply introduce the implementation idea?

It requires some implementation effort to the current fragment builder. The basic idea is to extend the original CSR/CSC for outgoing/incoming edges to a new one by adding new edge records. There would be some memory copy but should be practical. We have submitted the task to the OSPP program.

[songqing]

OK, thank you very much. How about constructing a new small fragment for the incremental data and keeping the full fragment unchanged, and providing a virtual fragment for users containing the two fragments? In this way, there will be less memory copy and cpu cost.

[sighingnow]

Such a design requires a complex implementation for iterators to access the vertices and edges in the graph, which, would introduce runtime overhead when running graph applications.

Due to the complexity of partitioned/distributed CSR (there would be cross-partition edges), maintaining such a "diff" data structure require implementation effort as well.

[songqing]

OK, I see, thanks a lot. Looking forward to the "OSPP program"'s production.

[sighingnow]

I'm going to move this thread to discussion.