Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

detect: log app-layer metadata in alert with single tx v5 #12168

Closed
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
detect: log app-layer metadata in alert with single tx
Ticket: 7199

Uses a config parameter detect.force-applayer-findtx to enable
this behavior (off by default)

This feature is requested for use cases with signatures not
using app-layer keywords but still targetting application
layer transactions, such as pass/drop rule combination,
or lua usage.

This overrides the previous behavior of checking if the signature
has a content match, by checking if there is only one live
transaction, in addition to the config parameter being set.
catenacyber committed Nov 28, 2024
commit e5d50ce1e1316a4bdc73810c9d85f1e90c42428d
15 changes: 15 additions & 0 deletions doc/userguide/output/eve/eve-json-output.rst
Original file line number Diff line number Diff line change
@@ -70,6 +70,21 @@ Alerts are event records for rule matches. They can be amended with
metadata, such as the application layer record (HTTP, DNS, etc) an
alert was generated for, and elements of the rule.

The alert is amended with application layer metadata for signatures
using application layer keywords. It is also the case for protocols
over UDP as each single packet is expected to contain a PDU.

For other signatures, the option ``detect.force-applayer-findtx``
can be used to force the detect engine to find a transaction.
These signatures need to define an app-layer protocol.
This transaction is not guaranteed to be the relevant one,
depending on your use case and how you define relevant here.
If there are multiple live transactions, none will get
picked up.
The alert event will have ``"tx_id_forced": true`` to recognize
these alerts.
Comment on lines +84 to +85
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pondering on the name here (as it will be used in the log output).
tx_match_forced? tx_logged_forced?

Copy link
Member

@jasonish jasonish Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

match has too much meaning with rule matching in Suricata-language, so I'd avoid that.



Metadata::

- alert:
4 changes: 4 additions & 0 deletions doc/userguide/upgrade.rst
Original file line number Diff line number Diff line change
@@ -96,6 +96,10 @@ Logging changes
~~~~~~~~~~~~~~~
- RFB security result is now consistently logged as ``security_result`` when it was
sometimes logged with a dash instead of an underscore.
- Application layer metadata is logged with alerts by default only for rules that
use application layer keywords. For other rules, the configuration parameter
catenacyber marked this conversation as resolved.
Show resolved Hide resolved
``detect.force-applayer-findtx`` can be used to force the detect engine to find a
transaction, which is not guaranteed to be the one you expect.

Upgrading 6.0 to 7.0
--------------------
4 changes: 4 additions & 0 deletions etc/schema.json
Original file line number Diff line number Diff line change
@@ -115,6 +115,10 @@
"tx_id": {
"type": "integer"
},
"tx_id_forced": {
"description": "the transaction for this alert event was forced, so the transaction may not be the one you expect",
catenacyber marked this conversation as resolved.
Show resolved Hide resolved
"type": "boolean"
},
"files": {
"type": "array",
"minItems": 1,
2 changes: 2 additions & 0 deletions src/decode.h
Original file line number Diff line number Diff line change
@@ -257,6 +257,8 @@ typedef struct PacketAlert_ {
#define PACKET_ALERT_RATE_FILTER_MODIFIED 0x10
/** alert is in a frame, frame_id set */
#define PACKET_ALERT_FLAG_FRAME 0x20
/** alert in a tx was forced */
#define PACKET_ALERT_FLAG_TX_FORCED 0x040

extern uint16_t packet_alert_max;
#define PACKET_ALERT_MAX 15
6 changes: 6 additions & 0 deletions src/detect-engine.c
Original file line number Diff line number Diff line change
@@ -2928,6 +2928,12 @@ static int DetectEngineCtxLoadConf(DetectEngineCtx *de_ctx)
"and 255, will default to 4");
}
}
int force_applayer = 0;
if ((ConfGetBool("detect.force-applayer-findtx", &force_applayer)) == 1) {
if (force_applayer == 1) {
de_ctx->force_applayer = true;
}
}

/* parse port grouping priority settings */

21 changes: 12 additions & 9 deletions src/detect.c
Original file line number Diff line number Diff line change
@@ -813,22 +813,25 @@ static inline void DetectRulePacketRules(
DetectRunPostMatch(tv, det_ctx, p, s);

uint64_t txid = PACKET_ALERT_NOTX;
if ((alert_flags & PACKET_ALERT_FLAG_STREAM_MATCH) ||
(s->alproto != ALPROTO_UNKNOWN && pflow->proto == IPPROTO_UDP)) {
// if there is a stream match (TCP), or
// a UDP specific app-layer signature,
// try to use the good tx for the packet direction
if (pflow->alstate) {
uint8_t dir =
(p->flowflags & FLOW_PKT_TOCLIENT) ? STREAM_TOCLIENT : STREAM_TOSERVER;
txid = AppLayerParserGetTransactionInspectId(pflow->alparser, dir);
if (pflow && pflow->alstate) {
uint8_t dir = (p->flowflags & FLOW_PKT_TOCLIENT) ? STREAM_TOCLIENT : STREAM_TOSERVER;
txid = AppLayerParserGetTransactionInspectId(pflow->alparser, dir);
if ((s->alproto != ALPROTO_UNKNOWN && pflow->proto == IPPROTO_UDP) ||
(de_ctx->force_applayer &&
AppLayerParserGetTxCnt(pflow, pflow->alstate) == txid + 1)) {
// if there is a UDP specific app-layer signature,
// or only one live transaction
// try to use the good tx for the packet direction
void *tx_ptr =
AppLayerParserGetTx(pflow->proto, pflow->alproto, pflow->alstate, txid);
AppLayerTxData *txd =
tx_ptr ? AppLayerParserGetTxData(pflow->proto, pflow->alproto, tx_ptr)
: NULL;
if (txd && txd->stream_logged < de_ctx->stream_tx_log_limit) {
alert_flags |= PACKET_ALERT_FLAG_TX;
if (pflow->proto != IPPROTO_UDP) {
alert_flags |= PACKET_ALERT_FLAG_TX_FORCED;
}
txd->stream_logged++;
}
}
3 changes: 3 additions & 0 deletions src/detect.h
Original file line number Diff line number Diff line change
@@ -887,6 +887,9 @@ typedef struct DetectEngineCtx_ {
/* maximum number of times a tx will get logged for a stream-only rule match */
uint8_t stream_tx_log_limit;

/* force app-layer tx finding for alerts with signatures not having app-layer keywords */
uint8_t force_applayer;

/* registration id for per thread ctx for the filemagic/file.magic keywords */
int filemagic_thread_ctx_id;

3 changes: 3 additions & 0 deletions src/output-json-alert.c
Original file line number Diff line number Diff line change
@@ -221,6 +221,9 @@ void AlertJsonHeader(const Packet *p, const PacketAlert *pa, JsonBuilder *js, ui
if (pa->flags & PACKET_ALERT_FLAG_TX) {
jb_set_uint(js, "tx_id", pa->tx_id);
}
if (pa->flags & PACKET_ALERT_FLAG_TX_FORCED) {
jb_set_bool(js, "tx_id_forced", true);
}

jb_open_object(js, "alert");

4 changes: 4 additions & 0 deletions suricata.yaml.in
Original file line number Diff line number Diff line change
@@ -1702,6 +1702,10 @@ detect:
inspection-recursion-limit: 3000
# maximum number of times a tx will get logged for a stream-only rule match
# stream-tx-log-limit: 4
# try to find an app-layer transaction for rules without app-layer keywords
# allows to log app-layer metadata in alert
# but the transaction may not be the relevant one.
# force-applayer-findtx: no
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

force-tx-log?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah... I am not sure.
This is the detect engine doing the work here, not the logging...

# If set to yes, the loading of signatures will be made after the capture
# is started. This will limit the downtime in IPS mode.
#delayed-detect: yes