Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log-parsing and standardize JSON output #24

Merged

Conversation

Andi-A
Copy link
Contributor

@Andi-A Andi-A commented Dec 1, 2023

This PR essentially adds 2 things:

  1. Event Log parsing and searching so only matching events are returned instead of the entire file. This is done by adding the --log_type argument, right now just supporting a value of "cloudtrail" but with commented stubs for other log formats. --log_type basically sets defined values for 2 other arguments --log_format ("json") and --log_properties (["Records"]), and these two arguments can also be set directly. This way users can on-the-fly define ANY log format and property list that cloudgrep will dive into to finally retrieve a LIST of logs that the cloudgrep search will be applied against without needing to add any more code.

  2. More formal output format. All results from cloudgrep are now JSON objects (json.dump of dict objects containing matched lines) without any printing of non-result strings. Any other output has been changed to logging.info or logging.warning. This allows cloudgrep results in stdout to be programmatically parsed by any calling code without the pipeline being polluted with informational printlines.

@chrisdoman
Copy link
Contributor

Excellent - PR looks great. I've made a couple of edits (below) and merged it 🎉

  • I've added some test coverage
  • Made the json output a flag, as I think most people will expect non json output by default, coming from grep
    Let me know if any of it doesn't look good, or doesn't work as expected following the changes.

@chrisdoman chrisdoman merged commit 8e09b81 into cado-security:main Dec 4, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants