-
Notifications
You must be signed in to change notification settings - Fork 181
Add support for pooling in codecs, partition_table, view etc. #445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
87c2b2c to
d7194bc
Compare
paulhenke
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Therefore we need to be able to tell the Codec when the message processing is done and the reference can be reused.
So codecs become stateful? Or were there already before?
| type nullCloser struct{} | ||
|
|
||
| func (n *nullCloser) Close() error { return nil } | ||
|
|
||
| var NoopCloser = new(nullCloser) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this for, if all the code calling DecodeP is doing a nil check on the io.Closer return.
|
Thanks @paulhenke for all your comments. As said in some comments, the idea of this PR is to draft how pooling could be supported, although goka does not actually use it, only the clients. The codecs would become stateful in some sense, that's true, but they haven't been before and don't have to be. But breaking the API for this is probably a bad idea, so we'll design it in a way that both ways are supported. |
d6a8f5a to
c1f7942
Compare
Work in Progress
This is a draft for evaluating the efficiency and consequences of adding (memory) pooling to the goka API.
How it works
When processing messages with goka
Processors, messages are decoded using aCodecin every callback call, as well as for everyJoinorLookup. The same happens on everyView.Get. When using more complex codecs like protobuf, those allocations add up and create pressure in the garbage collector. We would like to try reusing the messages. Protobuf supportsReseting the message upon unmarshalling, so they're a god fit for pooling.Therefore we need to be able to tell the
Codecwhen the message processing is done and the reference can be reused.This PR adds a new decode-function called
DecodePwhich returns anio.Closerinterface that must be called when the message can be reused.Additionally, it also adds a
GetPmethod to thestorage.Storageinterface to support pooling on disk-level, especially for Views.