Stream / batch hook processing for updates

[alechenninger]

Right now it looks like updates will pull many documents into memory at once. A heap dump shows 4000. Because updates pull in full documents, if these documents are large, it can quickly use up more than available heap, causing outages.

A client can work around by first retrieving ids of documents which match the query, and the iterate through them in smaller batches to apply the update. I don't see any reason lightblue couldn't do this server side.

[alechenninger]

I suspect this would require streaming responses as updates are processed (since the heap is collected as apart of the OperationContext document list I assume is used for the respose).

[alechenninger]

Another way to free up some memory for update responses may be to avoid keeping the original (full) documents around in memory longer than we need to.

[bserdar]

The main problem there is the hook processing. If we change hook processing to work on individual documents (or batches of documents) instead of lists of documents, we can process and throw away the updated docs.

…

On Tue, Jul 18, 2017 at 12:28 PM, Alec Henninger ***@***.***> wrote: Another way to free up some memory for update responses is to avoid keeping the original (full) documents around in memory longer than we need to. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#378 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADgDDR1Ukce96vjsPczWqGwKwRDoJ4UTks5sPPk8gaJpZM4Obtr0> .