Ive found that sites are less likely to block you from scraping them if you vary the user agent you send in your scrape requests. Being able to set 1 agent is good, but being able to set 10 or 20 and have the application choose one at random when requesting a target url would help reduce the chances of being blocked.
Ive found that sites are less likely to block you from scraping them if you vary the user agent you send in your scrape requests. Being able to set 1 agent is good, but being able to set 10 or 20 and have the application choose one at random when requesting a target url would help reduce the chances of being blocked.