Fasttable: added support for varint fields. #24163
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This involved a significant refactoring of the shared code in `cardinality.h`. This also makes a significant change to the design: we now allow the fast decoder to handle arena allocation fallback paths, whereas before the plan was to delegate these to the MiniTable decoder. While writing this CL, I realized that the previous plan was not feasible, at least not if we ever want to handle streaming decode. The problem is that a delimited field could require allocating memory, but if allocation fails in a subsequent buffer, we cannot back up to the previous buffer to fall back to the MiniTable decoder. If we use `preserve_most` for fallback paths, then handling allocation failure in the fast decoder should not be too bad. PiperOrigin-RevId: 768510853
|
Auto-closing Copybara pull request |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fasttable: added support for varint fields.
This involved a significant refactoring of the shared code in
cardinality.h.This also makes a significant change to the design: we now allow the fast decoder to handle arena allocation fallback paths, whereas before the plan was to delegate these to the MiniTable decoder. While writing this CL, I realized that the previous plan was not feasible, at least not if we ever want to handle streaming decode. The problem is that a delimited field could require allocating memory, but if allocation fails in a subsequent buffer, we cannot back up to the previous buffer to fall back to the MiniTable decoder.
If we use
preserve_mostfor fallback paths, then handling allocation failure in the fast decoder should not be too bad.