Broken link checker for internal links

[tleyden]

I'm planning to build a broken link checker on top of crawlee-python, similar in scope to lychee and htmltest. For now I just need to detect broken internal links, so the crawl should contain all of the state needed. By "internal links" I mean links within the page or to other pages within the same domain, including anchor fragments.

From reading through the docs:

• The BeautifulSoupCrawler seems like a great starting point since it already parses the HTML and makes it easy to extract links.
• The KeyValueStore seems like the right tool to store things like paths to detect internal broken links, and element IDs to detect broken anchor fragments.

Does crawlee already have a mechanism to detect internal broken links, or even broken anchor fragments? I'm still ramping up on crawlee, so I don't want to reinvent the wheel here.

Are there any other components or prior art I should be aware of?

PS: Thank you for releasing this as open source! From my experience so far, crawlee-python feels like a really powerful tool and it was extremely easy to get started with it.

[vdusek]

Hi,

glad to hear that, thanks for the nice words 🙂.

Regarding your question:

• For the crawler, you can use either BeautifulSoupCrawler or ParselCrawler, depending on your preference for HTML parser.
• To extract links, there is the enqueue_links helper method, available in both the BeautifulSoup and Parsel crawling contexts. It will do the links extraction for you. It supports different enqueue strategies, see EnqueueStrategy. You can find more details and an example here: Crawl a website with relative links.
• For storing results, you can use either the Dataset or the Key-value store. Check out the Result storage guide for more information.
• Crawlee does not have built-in mechanisms to detect broken links or anchor fragments.

I hope this helps and good luck with your project.