You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When starting 2 crawlers (A and B) to crawl 2 URLs with setTotalCrawlLimit(100) at the same time. The queues share the same total amount of crawl (100) and sometimes crawler A crawls URL of crawler B and vice versa.
That behavior doesn't happen when using ArrayCrawlQueue queue (the default one).
Moreover, when calling getProcessedUrlCount method, it always returns a negative number.
The text was updated successfully, but these errors were encountered:
When starting 2 crawlers (A and B) to crawl 2 URLs with
setTotalCrawlLimit(100)
at the same time. The queues share the same total amount of crawl (100) and sometimes crawler A crawls URL of crawler B and vice versa.That behavior doesn't happen when using
ArrayCrawlQueue
queue (the default one).Moreover, when calling
getProcessedUrlCount
method, it always returns a negative number.The text was updated successfully, but these errors were encountered: