feat(document_loaders): add exclude_urls to SitemapLoader

[CaillaudPA]

Description

This PR adds an exclude_urls parameter to the SitemapLoader class, allowing users to specify patterns for URLs that should be excluded from loading. This feature is particularly valuable for incremental content synchronization scenarios.

Key Use Case: Docusaurus Incremental Sync

When working with Docusaurus documentation sites, a common need is to only sync new or updated pages while skipping already processed content. This feature enables efficient incremental syncing by:

1. Maintaining a list of already processed URLs
2. Using exclude_urls to skip those URLs in subsequent runs
3. Only loading and processing new or modified content

Example usage with DocusaurusLoader:

# Store previously processed URLs
processed_urls = ["docs/tutorial/.*", "docs/api/v1/.*"]

# Only load new content
loader = DocusaurusLoader(
    "https://my-docs.com",
    exclude_urls=processed_urls,
    continue_on_failure=True
)

Benefits

• Reduced Processing Time: Skip already processed pages
• Lower Resource Usage: Avoid redundant content downloads
• Efficient Updates: Focus only on new or modified content
• Flexible Pattern Matching: Use regex patterns for precise control

Implementation Details

• Added exclude_urls parameter that accepts a list of regex patterns
• Patterns are checked after domain restrictions and filter patterns
• URLs matching any exclude pattern are skipped
• Maintains backward compatibility with existing usage

Testing

Added integration tests to verify:

• Basic URL exclusion functionality
• Interaction between filter and exclude patterns
• Edge cases with domain restrictions

Documentation

Updated docstrings and comments to explain:

• Parameter usage and behavior
• Regex pattern interpretation
• Example use cases

[eyurtsev]

Is there a way to create a unit test so the testing can happen on every PR? in langchain-community integrations tests aren't run as part of CI