Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib: user provided threads and packets - v9 #12401

Draft
wants to merge 20 commits into
base: master
Choose a base branch
from

Conversation

jasonish
Copy link
Member

Rebase of: #11921

Changes:

  • Fix memory leak in stats in refactor of acquisition loop

amirabell and others added 20 commits January 14, 2025 14:31
Add library source and runmode modules. Reorganized
library example to create a worker thread and replay a pcap
file using the library mode.
No API layer is added at this stage.

Edits by Jason Ish:
- fix guard
- add copyright/license headers
To keep the simple example simple, move the lib based capture method
example to its own example.
- the generated binaries for lib examples
- LSP files
- man pages
Worker threads not created by Suricata, but instead a library user
should not be joined, as Suricata does not have access to their thread
handle, and it may in-fact be an unjoinable thread, such as the main
process.

When the thread ID is 0, assume the thread is "externally" managed,
but still mark is as dead to satisfy Suricata's view of the thread.
Use the more conventional "--" command line handling to separate the
arguments. The first set will be passed to Suricata, and the args
after "--" will be handled by the example. Currently this is a single
PCAP filename, but will be extended to a list of PCAP filenames.
Also use a proper return type (ThreadVars *).
Update ThreadVars creation in lib mode to have the worker_id provided
by the user.
In the library capture example, show how the packet counter can be
updated.
Refactor TmThreadsSlotPktAcqLoop for user provided thread by breaking
out the init and finish code into their own functions.

For user provided threads, Suricata should not "drive" the thread, but
the setup and finish code is the same.

The finish function is exported so it can be called by the user
application when its receive loop or equivalent is done.

Also remove obsolete comment.
Also remove function to set the library mode. This is easy enough to
do with SCRunmodeSet, and we don't want to add a specific setter for
each and every runmode.
Add a release packet callback where the action can be checked for drop.
This also allows us to remove the sleep, as the ThreadVars are now
guaranteed to be created before PostInit.
This was a one line wrapper around SCTmThreadsSlotPktAcqLoopFinish, so
library users can call that directly instead.
Instead of immediately entering shutdown, use SuricataMainLoop to wait
for the end of the pcap.
This is better handled directly in the library user program so the
user has more control of the Packet structure, before and after
handling.
- SCPacketSetReleasePacket
- SCPacketSetLiveDevice
- SCPacketSetDatalink
- SCPacketSetTime
- SCPacketSetSource

Prevents direct access by library users and provides more ABI
stability.
Instead of direct field access.
@jasonish jasonish requested review from victorjulien and a team as code owners January 15, 2025 17:18
@jasonish jasonish marked this pull request as draft January 15, 2025 17:19
tmm_modules[TMM_DECODELIB].ThreadDeinit = DecodeLibThreadDeinit;
tmm_modules[TMM_DECODELIB].cap_flags = 0;
tmm_modules[TMM_DECODELIB].flags = TM_FLAG_DECODE_TM;
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source-lib files only exist for this decoder, which is rather generic. The pcap one could be used. We just require a decoder. It might be better if the library user application registered a decoder, with easy access to register a default one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't know if we'll complete it in time for 8, but the #12277 work would get rid of the separate decode module.


TmThreadsSetFlag(tv, THV_RUNNING);
return 0;
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar idea to source-lib here... Maybe these should become utility functions, where the library user register its own runmodes, with utility functions to setup the generic case easily.

@suricata-qa
Copy link

Information: QA ran without warnings.

Pipeline 24228

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants