Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider adding a robots.txt disallow line for the add_to_wishlist URLs #27

Open
garyillyes opened this issue Jul 2, 2024 · 0 comments

Comments

@garyillyes
Copy link

While there's already a rel=nofollow added to all Add to Wishlist buttons and noindex otherwise, the same URLs can be discovered by crawlers elsewhere on the site or offsite, without the nofollow attribute. This causes problem for some sites because crawlers discover those URLs and, in some cases, crawl a few million of the add_to_wishlist URLs before realizing they're useless.

If possible, consider injecting a new rule in the robots.txt of the sites that use the plugin to generally disallow crawling of these URLs. Something like:

disallow: /*add_to_wishlist
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant