-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VRL function to normalize size units like MiB, KiB, etc. to bytes #49
Comments
This is a very nice idea! However I would make sure to use the correct unit definitions and symbols. From https://en.wikipedia.org/wiki/Byte#Multiple-byte_units
The proposed |
It's much more important to me that the feature exist at all so I don't have strong opinions about what to call the parameter. I do still think the base should always default to 2, because that's what most people, other than hard drive marketeers, mean whether they include an i in the abbreviation or not. Also it should be overridable regardless. Even though it might make one's inner pedant cringe to use base 10 for a unit that is specifically defined as binary, the added flexibility is not a bad thing. |
I figured out a workaround for this until a function is added to VRL. For any fields I want to be normalized, I put them in a nested object called .normbytes. For example:
Then at the end of my pipeline I have this transform, so that any events with a .normbytes field will get the subfields normalized and then attached to the top-level event:
I think this technique is general enough to allow for creating any user-defined functions needed. |
I would propose to call it Also, I'd like to add I've seen byte amounts formatted also as eg. |
I am interested in implementing this feature! |
A note for the community
Use Cases
I frequently come across log messages containing sizes using human-readable units such as MB, KB, GB, or MiB, KiB, GiB. Normalizing these to bytes is a prerequisite for comparison and statistical analysis.
Attempted Solutions
I can do this using an if statement but it adds a lot of repetitive code to every transformation where I need it.
Proposal
Add a
normalize_bytes
function to VRL that will take a string specifying a size in human readable capacity units and return an integer number of bytes. The function should:binary
) that when set to false uses multiples of 1000 instead of 1024References
No response
Version
0.22.2
The text was updated successfully, but these errors were encountered: