Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NGINX Log Parsing Issue in Ubuntu 18.04.4 (missing evt.StrTime) #817

Open
oyale opened this issue Aug 31, 2023 · 3 comments
Open

NGINX Log Parsing Issue in Ubuntu 18.04.4 (missing evt.StrTime) #817

oyale opened this issue Aug 31, 2023 · 3 comments

Comments

@oyale
Copy link

oyale commented Aug 31, 2023

What happened?

After installing CrowdSec v1.5.2 on Ubuntu 18.04.4 using the provided repository and script, and then installing crowdsec and crowdsec-firewall-bouncer-iptables, the NGINX logs are not being parsed correctly.

What did you expect to happen?

  1. Install CrowdSec on Ubuntu using the repository.
  2. Install crowdsec and crowdsec-firewall-bouncer-iptables.
  3. Attempt to parse NGINX logs.

How can we reproduce it (as minimally and precisely as possible)?

I used the cscli explain command with the following log:

cscli explain --log '185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"' --type nginx --verbose

And received the following error:

WARN[31-08-2023 15:02:41] Line 0/1 is missing evt.StrTime. It is most likely a mistake as it will prevent your logs to be processed in time-machine/forensic mode. 
line: 185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> 185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
	|		├ update evt.Line.Src :  -> /tmp/cscli_explain4205633973/cscli_test_tmp.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-08-31 13:02:38.290108895 +0000 UTC
	|		├ create evt.Line.Labels.type : nginx
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : 185.132.xx.xxx - - [Aug/31/2023:13:57:03 +0200] "GET /cable HTTP/1.1" 101 24 6.890 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
	|		├ create evt.Parsed.program : nginx
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-08-31 13:02:38.290168537 +0000 UTC
	|		├ create evt.Meta.datasource_type : file
	|		├ create evt.Meta.datasource_path : /tmp/cscli_explain4205633973/cscli_test_tmp.log
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/nginx-logs
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

Anything else we need to know?

nginx's log_format:

log_format nginx '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent $request_time ' '"$http_referer" "$http_user_agent"';

nginx -v

nginx/1.18.0

Crowdsec version

$ cscli version
2023/08/31 14:49:38 version: v1.5.2-debian-pragmatic-4fbc3402fba932c8bd34b671527dcf7909d264c0
2023/08/31 14:49:38 Codename: alphaga
2023/08/31 14:49:38 BuildDate: 2023-05-26_16:13:25
2023/08/31 14:49:38 GoVersion: 1.20.1
2023/08/31 14:49:38 Platform: linux
2023/08/31 14:49:38 Constraint_parser: >= 1.0, <= 2.0
2023/08/31 14:49:38 Constraint_scenario: >= 1.0, < 3.0
2023/08/31 14:49:38 Constraint_api: v1
2023/08/31 14:49:38 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
$ uname -a
Linux server-redacted 4.15.0-196-generic crowdsecurity/crowdsec#207-Ubuntu SMP Thu Oct 27 21:24:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Enabled collections and parsers

$ cscli hub list -o raw
crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections
crowdsecurity/http-cve,enabled,2.1,,collections
crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections
crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/nginx-logs,enabled,1.4,Parse nginx access and error logs,parsers
crowdsecurity/sshd-logs,enabled,2.2,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers
crowdsecurity/CVE-2019-18935,enabled,0.1,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios
crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios
crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios
crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,0.8,Detect bad user-agents,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios
crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios
crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios
crowdsecurity/netgear_rce,enabled,0.2,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios
crowdsecurity/nginx-req-limit-exceeded,enabled,0.1,Detects IPs which violate nginx's user set request limit.,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios
ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios

Acquisition config

# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
#Generated acquisition file - wizard.sh (service: nginx) / files : /var/log/nginx/access.log /var/log/nginx/error.log
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx
---
#Generated acquisition file - wizard.sh (service: ssh) / files : /var/log/auth.log
filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog
filenames:
  - /var/log/syslog
labels:
  type: syslog
---
cat: '/etc/crowdsec/acquis.d/*': No such file or directory

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
  - Hub Folder              : /etc/crowdsec/hub
API Client:
  - URL                     : http://127.0.0.1:8080/
  - Login                   : bd854b54992945c7a769a8b2e2db5c77in5EoS25l5kmhqy0
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 127.0.0.1:8080
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

$ cscli metrics

Acquisition Metrics:
╭────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│             Source             │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/auth.log         │ 173        │ 3            │ 170            │ 10                     │
│ file:/var/log/nginx/access.log │ 574        │ -            │ 574            │ -                      │
│ file:/var/log/syslog           │ 34         │ -            │ 34             │ -                      │
╰────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯

Bucket Metrics:
╭─────────────────────────────────────┬───────────────┬───────────┬──────────────┬────────┬─────────╮
│               Bucket                │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├─────────────────────────────────────┼───────────────┼───────────┼──────────────┼────────┼─────────┤
│ crowdsecurity/ssh-bf                │ -             │ -         │ 2            │ 3      │ 2       │
│ crowdsecurity/ssh-bf_user-enum      │ -             │ -         │ 2            │ 2      │ 2       │
│ crowdsecurity/ssh-slow-bf           │ -             │ -         │ 2            │ 3      │ 2       │
│ crowdsecurity/ssh-slow-bf_user-enum │ -             │ -         │ 2            │ 2      │ 2       │
╰─────────────────────────────────────┴───────────────┴───────────┴──────────────┴────────┴─────────╯

Parser Metrics:
╭─────────────────────────────────┬───────┬────────┬──────────╮
│             Parsers             │ Hits  │ Parsed │ Unparsed │
├─────────────────────────────────┼───────┼────────┼──────────┤
│ child-crowdsecurity/nginx-logs  │ 1.15k │ -      │ 1.15k    │
│ child-crowdsecurity/sshd-logs   │ 158   │ 3      │ 155      │
│ child-crowdsecurity/syslog-logs │ 207   │ 207    │ -        │
│ crowdsecurity/dateparse-enrich  │ 3     │ 3      │ -        │
│ crowdsecurity/geoip-enrich      │ 3     │ 3      │ -        │
│ crowdsecurity/nginx-logs        │ 574   │ -      │ 574      │
│ crowdsecurity/non-syslog        │ 574   │ 574    │ -        │
│ crowdsecurity/sshd-logs         │ 16    │ 3      │ 13       │
│ crowdsecurity/syslog-logs       │ 207   │ 207    │ -        │
│ crowdsecurity/whitelists        │ 3     │ 3      │ -        │
╰─────────────────────────────────┴───────┴────────┴──────────╯

Local Api Metrics:
╭──────────────────────┬────────┬──────╮
│        Route         │ Method │ Hits │
├──────────────────────┼────────┼──────┤
│ /v1/decisions/stream │ GET    │ 432  │
│ /v1/heartbeat        │ GET    │ 72   │
│ /v1/watchers/login   │ POST   │ 9    │
╰──────────────────────┴────────┴──────╯

Local Api Machines Metrics:
╭──────────────────────────────────────────────────┬───────────────┬────────┬──────╮
│                     Machine                      │     Route     │ Method │ Hits │
├──────────────────────────────────────────────────┼───────────────┼────────┼──────┤
│ bd854b54992945c7a769a8b2e2db5c77in5EoS25l5kmhqy0 │ /v1/heartbeat │ GET    │ 72   │
╰──────────────────────────────────────────────────┴───────────────┴────────┴──────╯

Local Api Bouncers Metrics:
╭────────────────────────────────┬──────────────────────┬────────┬──────╮
│            Bouncer             │        Route         │ Method │ Hits │
├────────────────────────────────┼──────────────────────┼────────┼──────┤
│ cs-firewall-bouncer-1693480692 │ /v1/decisions/stream │ GET    │ 432  │
╰────────────────────────────────┴──────────────────────┴────────┴──────╯

Local Api Decisions:
╭────────────────────────────────────────────┬────────┬────────┬───────╮
│                   Reason                   │ Origin │ Action │ Count │
├────────────────────────────────────────────┼────────┼────────┼───────┤
│ crowdsecurity/fortinet-cve-2018-13379      │ CAPI   │ ban    │ 97    │
│ crowdsecurity/http-generic-bf              │ CAPI   │ ban    │ 11    │
│ crowdsecurity/nginx-req-limit-exceeded     │ CAPI   │ ban    │ 131   │
│ firehol_botscout_7d                        │ lists  │ ban    │ 6067  │
│ crowdsecurity/CVE-2019-18935               │ CAPI   │ ban    │ 36    │
│ crowdsecurity/spring4shell_cve-2022-22965  │ CAPI   │ ban    │ 1     │
│ crowdsecurity/thinkphp-cve-2018-20062      │ CAPI   │ ban    │ 29    │
│ crowdsecurity/CVE-2022-26134               │ CAPI   │ ban    │ 150   │
│ crowdsecurity/CVE-2022-41082               │ CAPI   │ ban    │ 867   │
│ crowdsecurity/CVE-2022-42889               │ CAPI   │ ban    │ 14    │
│ crowdsecurity/grafana-cve-2021-43798       │ CAPI   │ ban    │ 69    │
│ crowdsecurity/http-path-traversal-probing  │ CAPI   │ ban    │ 56    │
│ crowdsecurity/http-sensitive-files         │ CAPI   │ ban    │ 7     │
│ crowdsecurity/CVE-2022-35914               │ CAPI   │ ban    │ 49    │
│ crowdsecurity/http-backdoors-attempts      │ CAPI   │ ban    │ 795   │
│ crowdsecurity/netgear_rce                  │ CAPI   │ ban    │ 23    │
│ crowdsecurity/CVE-2022-37042               │ CAPI   │ ban    │ 20    │
│ crowdsecurity/http-cve-2021-41773          │ CAPI   │ ban    │ 13    │
│ crowdsecurity/http-bad-user-agent          │ CAPI   │ ban    │ 5458  │
│ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI   │ ban    │ 372   │
│ crowdsecurity/http-probing                 │ CAPI   │ ban    │ 2003  │
│ crowdsecurity/jira_cve-2021-26086          │ CAPI   │ ban    │ 20    │
│ crowdsecurity/ssh-bf                       │ CAPI   │ ban    │ 4043  │
│ crowdsecurity/ssh-slow-bf                  │ CAPI   │ ban    │ 8     │
│ crowdsecurity/http-open-proxy              │ CAPI   │ ban    │ 292   │
│ crowdsecurity/http-crawl-non_statics       │ CAPI   │ ban    │ 415   │
│ crowdsecurity/f5-big-ip-cve-2020-5902      │ CAPI   │ ban    │ 21    │

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

No response
@github-actions
Copy link

@oyale: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

@oyale oyale changed the title NGINX Log Parsing Issue in Ubuntu 18.04.4 NGINX Log Parsing Issue in Ubuntu 18.04.4 (missing evt.StrTime) Aug 31, 2023
@oyale
Copy link
Author

oyale commented Aug 31, 2023 •
edited
Loading

I've realized that the difference was the response time in the log. I've managed to temporally fix it by adding:

  RESPONSETIME: '[0-9]+\.[0-9]+'

to the pattern_syntax and modifying the grok pattern by adding it:

      pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - %{NGCUSTOMUSER:remote_user}? \[%{HTTPDATE:time_local}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} (%{RESPONSETIME:response_time})? "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'

(both of them on /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml).

I wonder what is the proper way to fix the expression because I guess those patches will be overridden in future updates.

Could you help me with that? Should I write a new parser?

@LaurenceJJones
Copy link
Contributor

Never knew the response time could be a float, I will transfer to the hub which can handle this update.

@LaurenceJJones LaurenceJJones transferred this issue from crowdsecurity/crowdsec Sep 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@LaurenceJJones @oyale