Cloudflare Protection Blocking Crunchbase Results

[PatricioCabo]

Is your feature request related to a problem? Please describe.
When trying to scrape Crunchbase pages through Tavily, the results are being blocked by Cloudflare's security protection. This affects the quality of company research as Crunchbase is a crucial source for company information.

Current Behavior

• Tavily attempts to scrape Crunchbase URLs (e.g., "https://www.crunchbase.com/organization/XXXXX")
• Instead of getting company data, we receive Cloudflare's block page:

  "please enable cookies. sorry, you have been blocked you are unable to access crunchbase.com..."
  

• This causes our similarity check to fail as the content doesn't contain company information

Describe the solution you'd like

• Tavily should be able to bypass Cloudflare protection
• Successfully retrieve company information from Crunchbase pages

Describe alternatives you've considered

• Implement headless browser support in Tavily (e.g., Playwright, Puppeteer)
• Add proper headers and cookie management

[ElishaKay]

Welcome @PatricioCabo

Agreed.
It would be great if someone from the community would take a crack at adding Tavily Extract functionality to GPTR to avoid the scraper getting blocked.

Green light for the PR to whoever feels the calling