Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds detection for various bots #7480

Merged
merged 7 commits into from
Oct 6, 2023
Merged

Adds detection for various bots #7480

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6b871ea
Adds detection for phpMyAdmin
liviuconcioiu Oct 6, 2023
6394f95
Adds detection for Cloudflare SSL Detector
liviuconcioiu Oct 6, 2023
f6ca34d
Change producer url for Cloudflare Health Checks
liviuconcioiu Oct 6, 2023
be816e4
Improves detection for Cloudflare Diagnostics
liviuconcioiu Oct 6, 2023
5221e6b
Adds detection for Cloudflare Custom Hostname Verification
liviuconcioiu Oct 6, 2023
bebd728
Adds detection for Cloudflare Traffic Manager
liviuconcioiu Oct 6, 2023
2eb7040
Adds detection for Cloudflare Security Insights
liviuconcioiu Oct 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 55 additions & 4 deletions Tests/fixtures/bots.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -584,10 +584,46 @@
bot:
name: CloudFlare Always Online
category: Site Monitor
url: http://www.cloudflare.com/always-online
url: https://www.cloudflare.com/always-online
producer:
name: CloudFlare
url: http://www.cloudflare.com
url: https://www.cloudflare.com/
-
user_agent: Cloudflare-SSLDetector
bot:
name: Cloudflare SSL Detector
category: Site Monitor
url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/
producer:
name: CloudFlare
url: https://www.cloudflare.com/
-
user_agent: Cloudflare Custom Hostname Verification
bot:
name: Cloudflare Custom Hostname Verification
category: Service Agent
url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/
producer:
name: CloudFlare
url: https://www.cloudflare.com/
-
user_agent: 'Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: <POOLID>)'
bot:
name: Cloudflare Traffic Manager
category: Site Monitor
url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/
producer:
name: CloudFlare
url: https://www.cloudflare.com/
-
user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)
bot:
name: Cloudflare Security Insights
category: Site Monitor
url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/
producer:
name: CloudFlare
url: https://www.cloudflare.com/
-
user_agent: collectd/5.5.1
bot:
Expand Down Expand Up @@ -3926,7 +3962,16 @@
url: https://www.cloudflare.com/
producer:
name: Cloudflare
url: https://www.cloudflare.com
url: https://www.cloudflare.com/
-
user_agent: Cloudflare-diagnostics
bot:
name: Cloudflare Diagnostics
category: Site Monitor
url: https://www.cloudflare.com/
producer:
name: Cloudflare
url: https://www.cloudflare.com/
-
user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) adbeat.com/policy AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/69.0.3497.105 Mobile/15E148 Safari/605.1
bot:
Expand Down Expand Up @@ -5565,7 +5610,7 @@
url: https://developers.cloudflare.com/health-checks/
producer:
name: CloudFlare
url: http://www.cloudflare.com
url: https://www.cloudflare.com/
-
user_agent: Mozilla/5.0 (Morningscore/1.0)
bot:
Expand Down Expand Up @@ -5777,3 +5822,9 @@
producer:
name: webwiki GmbH
url: https://www.webwiki.com/
-
user_agent: phpMyAdmin
bot:
name: phpMyAdmin
category: Service Agent
url: https://www.phpmyadmin.net/
47 changes: 42 additions & 5 deletions regexes/bots.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
url: 'https://developers.cloudflare.com/health-checks/'
producer:
name: 'CloudFlare'
url: 'http://www.cloudflare.com'
url: 'https://www.cloudflare.com/'

- regex: '360Spider'
name: '360Spider'
Expand Down Expand Up @@ -331,21 +331,53 @@
name: 'CloudFlare'
url: 'http://www.cloudflare.com'

- regex: 'CloudflareDiagnostics'
- regex: 'Cloudflare-?Diagnostics'
name: 'Cloudflare Diagnostics'
category: 'Site Monitor'
url: 'https://www.cloudflare.com/'
producer:
name: 'Cloudflare'
url: 'https://www.cloudflare.com'
url: 'https://www.cloudflare.com/'

- regex: 'CloudFlare-AlwaysOnline'
name: 'CloudFlare Always Online'
category: 'Site Monitor'
url: 'http://www.cloudflare.com/always-online'
url: 'https://www.cloudflare.com/always-online'
producer:
name: 'CloudFlare'
url: 'http://www.cloudflare.com'
url: 'https://www.cloudflare.com/'

- regex: 'Cloudflare-SSLDetector'
name: 'Cloudflare SSL Detector'
category: 'Site Monitor'
url: 'https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/'
producer:
name: 'CloudFlare'
url: 'https://www.cloudflare.com/'

- regex: 'Cloudflare Custom Hostname Verification'
name: 'Cloudflare Custom Hostname Verification'
category: 'Service Agent'
url: 'https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/'
producer:
name: 'CloudFlare'
url: 'https://www.cloudflare.com/'

- regex: 'Cloudflare-Traffic-Manager'
name: 'Cloudflare Traffic Manager'
category: 'Site Monitor'
url: 'https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/'
producer:
name: 'CloudFlare'
url: 'https://www.cloudflare.com/'

- regex: 'https://developers.cloudflare.com/security-center/'
name: 'Cloudflare Security Insights'
category: 'Site Monitor'
url: 'https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/'
producer:
name: 'CloudFlare'
url: 'https://www.cloudflare.com/'

- regex: 'coccoc.com'
name: 'Cốc Cốc Bot'
Expand Down Expand Up @@ -3435,6 +3467,11 @@
name: 'webwiki GmbH'
url: 'https://www.webwiki.com/'

- regex: 'phpMyAdmin'
name: 'phpMyAdmin'
category: 'Service Agent'
url: 'https://www.phpmyadmin.net/'

# Generic detections
- regex: '[a-z0-9\-_]*((?<!cu|power[ _]|m[ _])bot(?![ _]TAB|[ _]?5[0-9]|[ _]Senior|[ _]Junior)|crawler|crawl|checker|archiver|transcoder|spider)([^a-z]|$)'
name: 'Generic Bot'
Loading