robots.txt and ai.txt ignored #19458
Schamschula
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
As a webmaster, I've found the agent string
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 (compatible; pytorch-lightning; +https://github.com/Lightning-AI/pytorch-lightning)"
for requests in my logs. However, there never was a corresponding request from the host torobots.txt
orai.txt
. Even worse, there are requests by"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0"
without the link to GitHub.com (perhaps an older version?). Unless I know what the data is used for I will block such agents from my site as scraping of graphics is not appreciated.Beta Was this translation helpful? Give feedback.
All reactions