With the following crawler configuration:

from bs4 import BeautifulSoup as Soup

url = "https://example.com"
loader = RecursiveUrlLoader(
    url=url, max_depth=2, extractor=lambda x: Soup(x, "html.parser").text 
)
docs = loader.load()

An attacker in control of the contents of https://example.com could place a malicious HTML file in there with links like "https://example.completely.different/my_file.html" and the crawler would proceed to download that file as well even though prevent_outside=True.

https://github.com/langchain-ai/langchain/blob/bf0b3cc0b5ade1fb95a5b1b6fa260e99064c2e22/libs/community/langchain_community/document_loaders/recursive_url_loader.py#L51-L51

Resolved in langchain-ai/langchain#15559

References

• https://nvd.nist.gov/vuln/detail/CVE-2024-0243
• langchain-ai/langchain@bf0b3cc
• https://huntr.com/bounties/370904e7-10ac-40a4-a8d4-e2d16e1ca861
• langchain-ai/langchain#15559
• https://github.com/langchain-ai/langchain/blob/bf0b3cc0b5ade1fb95a5b1b6fa260e99064c2e22/libs/community/langchain_community/document_loaders/recursive_url_loader.py#L51-L51