Plain text extraction methods

[prasantpoudel]

As of now there is only title and content extraction methods are there(may other also vailable), also we can implement another methods that extract the plain text after removing all the tags from the html.

[janbuchar]

Hello, and thank you for your interest in Crawlee! Sophisticated text extraction is out of scope of this project.

• If you just need to remove HTML tags completely, period, then you can code that in your request handlers using whatever facilities Playwright/Beautifulsoup/Parsel provides to you.
  • I have opened A helper function for simple HTML tag removal #659 for adding a simple helper that does this using BeautifulSoup.
• If you're looking to extract content from the pages, you might want to use something like crawl4ai. You can even integrate it with BasicCrawler from Crawlee.
• If you're looking for a complete text extraction solution, you can use the Website Content Crawler from Apify.