Bulk Support

[sankarp-kavach]

I have a few million records in my program that I want to bulk insert using my Go program. It takes me about 2 minutes for 500 records and that would take me 4000 hours for just a million records. I have a couple of MERGE statements in each iteration.

I read about how to improve this and stumbled on to the GRAPH.BULK which may seem useful in this case. But the go library does not seem to support this as the documentation is not covering it. Are there plans to support the BULK API for usecases similar to mine ? Inserting a million records (a few hundred nodes and edges) may not be such a rare scenario. Thanks.

[gkorland]

Is it a one thing? Meaning is it the only the initial load?
Please check https://github.com/RedisGraph/redisgraph-bulk-loader

[sankarp-kavach]

Is it a one thing? Meaning is it the only the initial load?
Please check https://github.com/RedisGraph/redisgraph-bulk-loader

Thanks. This is what I am evaluating now. But nevertheless it may be a good idea to implement support for the bulk APIs here. It will help in some cases where we get a few hundred graph edges that we would need to update, on the results of a batch job completion or some such. Thanks.

[gkorland]

Notice the current API is for Bulk-Load and not for Bulk-Update

[sankarp-kavach]

@gkorland Are there any other alternatives / workarounds to do operations in bulk ? In our system we have a tight loop where a lot of graph.Query calls have to happen and it is becoming a bottleneck. The pipelining support documented in https://redis.io/topics/pipelining seems interesting but the go library does not seem to support it. I can probably create a wrapper and append the queries with a \r\n manually and parse the results. But if it was in the library it would be less error prone. Thanks.

[sankarp-kavach]

@gkorland Will you be interested in getting a PR if we could contrib a QueryBulk API to which we give an array of strings, whose responses will be sent in bulk ? We want to avoid multiple trips between redis and our backend go server and want to batch some calls. There is no clean / easy way now.