Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

preRequest function cuts the entire branch instead of a single page #357

Open
AleksandrBorkun opened this issue Nov 5, 2019 · 0 comments

Comments

@AleksandrBorkun
Copy link

What is the current behavior?

preRequest function cutting a lot of links in case of URL regexp filtering

If the current behavior is a bug, please provide the steps to reproduce

const isVisitedMap = { '\\.com\\/users\\/[\\d]+[\\w]+': false //CONSUMER_PROFILE_REGEXP }

function isVisited(url){
  for(const [key, value] of Object.entries(isVisitedMap)){
      if(new RegExp(key, 'g').test(url)){
          if(value){
              return value;
          }else{
              isVisitedMap[key] = true;
              return false;
          }
      }
  }
  return false;
}

(async () => {
  const crawler = await HCCrawler.launch({
    // Function to be evaluated in browsers
    preRequest: (async (opt) => { return !isVisited(opt.url)}),
    evaluatePage: (() => ({
      pagePath: window.location.href,
    })),
    // Function to be called with evaluated results from browsers
    onSuccess: (result => {
      console.log('pagePath:', result.result.pagePath);
    }),
    exporter
  });

  // Queue a request
   await crawler.queue({
    url: 'https://www.example.com/',
    headless: false,
    maxDepth: 4,
    userAgent: 'DuckDuckBot',
    allowedDomains: [/example\.com$/],
    skipDuplicates: true,
  });
  await crawler.onIdle(); // Resolved when no queue is left
  await crawler.close(); // Close the crawler
})();

What is the expected behavior?
an only single page that match regexp skipped

What is the motivation / use case for changing the behavior?

if you'll try to crawl web sites with public access to the user's profile pages (or any other entities pages), you'll probably want to skip all of the user profile links except one, because all of them are similar.
I thought that I can skip a single page by returning false in preRequest function.

Please tell us about your environment:

  • Version: "headless-chrome-crawler": "^1.8.0",
  • Platform / OS version: Windows 10
  • Node.js version: v10.16.3 npm: '6.9.0',
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant