Detect integers 6644 v14

[catenacyber]

Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/6644 and all subtickets
https://redmine.openinfosecfoundation.org/issues/6645
https://redmine.openinfosecfoundation.org/issues/6646
https://redmine.openinfosecfoundation.org/issues/6647
https://redmine.openinfosecfoundation.org/issues/6648
https://redmine.openinfosecfoundation.org/issues/6628

Describe changes:

• detect/integers: support hexadecimal notation for parsing
• detect/integers: add mode for negated range
• detect/integers: rust derive for enumerations
• detect/integers: keywords now accept bitmasks
• doc: detect/integers

#10241 rebased to get green CI again

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (c3b3c11) 82.28% compared to head (77f0023) 82.28%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10246      +/-   ##
==========================================
- Coverage   82.28%   82.28%   -0.01%     
==========================================
  Files         977      978       +1     
  Lines      271950   272068     +118     
==========================================
+ Hits       223784   223873      +89     
- Misses      48166    48195      +29     

Flag	Coverage Δ
fuzzcorpus	63.38% <42.85%> (-0.02%)	⬇️
suricata-verify	61.48% <42.85%> (-0.04%)	⬇️
unittests	62.83% <97.61%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[jufajardini]

Doc changes look good to me.

Proposed changes are very welcome, imho.

[suricata-qa]

Information:

ERROR: QA failed on SURI_TLPW1_files_sha256.

field	baseline	test	%
	SURI_TLPR1_stats_chk
.app_layer.error.http.parser	1108	724	65.34%

Pipeline 17748

[zoomequipd]

hey @catenacyber I had an question about the bitmask operation that went unanswered in a previous PR. (sorry, just now catching up on this feature)

Currently bitmask operations supported by byte_test, byte_match, byte_jump (maybe not byte_jump??) does a right shift by the number of trailing 0 (zeros) after the AND operation. Does this bitmask option perform the same behavior?

it might be worth noting it if does or does not within the bitmask section of the docs.

[catenacyber]

hey @catenacyber I had an question about the bitmask operation that went unanswered in a previous PR. (sorry, just now catching up on this feature)

Currently bitmask operations supported by byte_test, byte_match, byte_jump (maybe not byte_jump??) does a right shift by the number of trailing 0 (zeros) after the AND operation. Does this bitmask option perform the same behavior?

it might be worth noting it if does or does not within the bitmask section of the docs.

@zoomequipd I had answered here #10179 (comment) ;-)

You can do bsize:&0xc0=0x80.

You should not do bsize:&0xc0=2.

I can add a note about the difference with byte_test

[victorjulien]

Merged in #10277, thanks!

[victorjulien]

hey @catenacyber I had an question about the bitmask operation that went unanswered in a previous PR. (sorry, just now catching up on this feature)

Currently bitmask operations supported by byte_test, byte_match, byte_jump (maybe not byte_jump??) does a right shift by the number of trailing 0 (zeros) after the AND operation. Does this bitmask option perform the same behavior?

it might be worth noting it if does or does not within the bitmask section of the docs.

@zoomequipd do you see a reason for us to adopt that same approach here, as opposed to keeping it as-is and documenting it?

[catenacyber]

Also, this bitmask for integer keywords is more like the "and" operator of byte_test ;-)

[zoomequipd]

@zoomequipd do you see a reason for us to adopt that same approach here, as opposed to keeping it as-is and documenting it?

I asked the newer rule writers with ET, they all agreed that consistency would be preferred.

While the right shift behavior is a bit of a nuance itself, it's at least used, currently, everywhere bitmask is used. Not using it would create yet another nuance specific to one keyword's implementation. As a rule writer it is difficult to remember and track such nuances, even when documented.

As such, it'd be our preference to be consistent and implement the right shift operation.

[victorjulien]

@zoomequipd do you see a reason for us to adopt that same approach here, as opposed to keeping it as-is and documenting it?

I asked the newer rule writers with ET, they all agreed that consistency would be preferred.

While the right shift behavior is a bit of a nuance itself, it's at least used, currently, everywhere bitmask is used. Not using it would create yet another nuance specific to one keyword's implementation. As a rule writer it is difficult to remember and track such nuances, even when documented.

As such, it'd be our preference to be consistent and implement the right shift operation.

Can you explain the use case of the right shift? I think we implemented it from the snort language, but I can't find any reasoning about it.

[catenacyber]

it's at least used, currently, everywhere bitmask is used.

It is not used in byte_test and operator. (If you wish, I can rename the detect/integer as "and operator")
And it is not used in byte_jump bitmask documentation (but not sure it works either)

[zoomequipd]

It is not used in byte_test and operator.

Correct? It's used with the 'bitmask' option, which i'm pretty sure can be applied to any operation, but is most frequently used with the =
https://redmine.openinfosecfoundation.org/issues/3283

(If you wish, I can rename the detect/integer as "and operator")

That'd be fine? If it's going to support the bitmask option, the bitmask option should support the right shift behavior. I will say, the bitmask option does come in hand when attempting to check values less than a whole byte.

And it is not used in byte_jump bitmask documentation (but not sure it works either)

byte_jump doesn't implement bitmask option at all https://redmine.openinfosecfoundation.org/issues/6693

Can you explain the use case of the right shift? I think we implemented it from the snort language, but I can't find any reasoning about it.

I cannot, I just work with what the tools provide.

[catenacyber]

That'd be fine? If it's going to support the bitmask option, the bitmask option should support the right shift behavior. I will say, the bitmask option does come in hand when attempting to check values less than a whole byte.

Disclaimer : I am not sure we speak about the same things...
I could add a "bitmask option" to the integer keywords. I do not see the use case it solves better than the and operator...