detect/entropy: Add entropy keyword #12539
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Issue: 4162 This commit adds a floating-point analog to `uint.rs` targeted initially to be used by the forthcoming entropy keyword.
This commit adds
- Parser for the entropy keyword
- Calculation of content the Shannon entropy value
Issue: 4162
The entropy keyword syntax is the keyword entropy followed by options
and the entropy value for comparison.
The minimum entropy keyword specification is:
entropy: value <entropy-spec>
This results in the calculated entropy value being compared with
<entropy-spec> with the equality operator.
A match occurs when the values and operator agree. This example matches
if the calculated and entropy value are the same.
When entropy keyword options are specified, all options and "value" must
be comma-separated. Options and value may be specified in any order.
Options have default values:
- bytes is equal to the current content length
- offset is 0
- comparison with value is equality
entropy: [bytes <byteval>] [offset <offsetval>] value <entropy-spec>
Using default values:
entropy: bytes 0, offset 0, value =<entropy-spec>
<entropy-spec> is: <operator> (see below) and a value, e.g., "< 4.1"
The following operators are available from the float crate:
- = (default): Match when calculated entropy value equals specified entropy value
- < Match when calculated entropy value is strictly less than specified entropy value
- <= Match when calculated entropy value is less than or equal to specified entropy value
- > Match when calculated entropy value is strictly greater than specified entropy value
- >= Match when calculated entropy value is greater than or equal to specified entropy value
- != Match when calculated entropy value is not equal to specified entropy value
- x-y Match when calculated entropy value is in the range, exclusive
- !x-y Match when calculated entropy value is not in the range, exclusive
This commit adds keyword/build support for the entropy keyword. The entropy keyword compares an entropy value with a value calculated according to the Shannon entropy on the available content. Issue: 4162
This commit updates the - Upgrade notes for 7 to 8 - Payload keyword section Both are update to document the new entropy keyword.
This commit causes the content inspection engine to recognize and invoke the entropy "match" function when the entropy keyword is used. Issue: 4162
jlucovsky
requested review from
jasonish,
jufajardini and
victorjulien
as code owners
This was referenced Feb 6, 2025
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12539 +/- ##
==========================================
+ Coverage 80.68% 80.72% +0.03%
==========================================
Files 925 928 +3
Lines 258914 259537 +623
==========================================
+ Hits 208914 209511 +597
- Misses 50000 50026 +26
Flags with carried forward coverage won't be shown. Click here to find out more. |
|
Information: QA ran without warnings. Pipeline 24649 |
jasonish
reviewed
Feb 6, 2025
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
| use assert_float_eq::*; |
There was a problem hiding this comment.
Just a little hesitation on adding another crate. Would have formatting to a string with 2 or 3 decimal points and comparing that have worked just as well?
There was a problem hiding this comment.
I'll work things so the crate's no longer required.
jasonish
reviewed
Feb 6, 2025
|
|
||
| This example shows all possible options with default values:: | ||
|
|
||
| entropy: bytes 0, offset 0, value =<entropy-val> |
There was a problem hiding this comment.
For an example, can we use a real value so we're not mixing in chars like < and > in the placeholder?
There was a problem hiding this comment.
Thanks ... will do.
|
Continued in #12542 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Continuation of #12385
The entropy keyword syntax is the keyword
entropyfollowed by optionsand the entropy value for comparison.
The minimum entropy keyword specification is:
entropy: value <entropy-spec>This results in the calculated entropy value being compared with
with the equality operator.
A match occurs when the values and operator agree. This example matches
if the calculated and entropy value are the same.
When entropy keyword options are specified, all options and "value" must
be comma-separated. Options and value may be specified in any order.
Options have default values:
entropy: [bytes <byteval>] [offset <offsetval>] value <entropy-spec>Using default values:
entropy: bytes 0, offset 0, value =<entropy-spec>is: (see below) and a value, e.g., "< 4.1"
The following operators are available from the float crate introduced with this pr:
- = (default): Match when calculated entropy value equals specified entropy value
- < Match when calculated entropy value is strictly less than specified entropy value
- <= Match when calculated entropy value is less than or equal to the specified entropy value
- > Match when calculated entropy value is strictly greater than specified entropy value
- >= Match when calculated entropy value is greater than or equal to the specified entropy value
- != Match when the calculated entropy value is not equal to the specified entropy value
- x-y Match when calculated entropy value is in the range, exclusive
- !x-y Match when calculated entropy value is not in the range, exclusive
Link to ticket: https://redmine.openinfosecfoundation.org/issues/4162
Describe changes:
Provide values to any of the below to override the defaults.
link to the pull request in the respective
_BRANCHvariable.SV_BRANCH=OISF/suricata-verify#2232