How to change User Agent in PlaywrightCrawler?

[LeMoussel]

How can I change the user_agent in PlaywrightCrawler?

Here's what I tried:

from crawlee.playwright_crawler import PlaywrightCrawler

crawler = PlaywrightCrawler(browser_options={'user_agent': "My User Agent"})

However, I encountered the following error: BrowserType.launch() got an unexpected keyword argument 'user_agent'.

[LeMoussel]

As described in PR #753 , I specific user_agent to use but I still have an error:

File "c:\Users\pc\AppData\Local\Programs\Python\Python310\lib\site-packages\crawlee\playwright_crawler\_playwright_crawler.py", line 108, in __init__
    super().__init__(**kwargs)
TypeError: BasicCrawler.__init__() got an unexpected keyword argument 'browser_options'

[janbuchar]

Hi @LeMoussel, the browser_options parameter is not yet released, so unless you are using a beta release, this works as expected.

When are we planning to make a new release @vdusek?

[Mantisus]

Hi @LeMoussel, the user_agent is not an option for browser_options. It must be passed in when the context is created. Watch out for #755

You can now set the User-Agent as the header for the request

But if your goal is to replace the default User-Agent in Playwright, note that this already happens automatically with HeaderGenerator

[LeMoussel]

It seems I'm missing something here. 🙂
When I set the User-Agent header for the request like this:

    import json

    from crawlee import Request
    from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext

    crawler = PlaywrightCrawler(
        max_requests_per_crawl=1,
    )

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f"Processing {context.request.url} ...")
        response = await context.response.text()
        data = json.loads(response)
        print(data['user-agent'])

    await crawler.run(
        [
            Request.from_url(
                url="https://httpbingo.org/user-agent",
                headers={"User-Agent": "Test User Agent"},
            )
        ]
    )

I get the following result:

[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 1; cpu = 0.0; mem = 0.0; event_loop = 1.0; client_info = 0.0
[crawlee.playwright_crawler._playwright_crawler] INFO  Processing https://httpbingo.org/user-agent ...
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
[crawlee.playwright_crawler._playwright_crawler] INFO  The crawler has reached its limit of 1 requests per crawl. All ongoing requests have now completed. Total requests processed: 1. The crawler will now shut down.
[crawlee._autoscaling.autoscaled_pool] INFO  Waiting for remaining tasks to finish
[crawlee.playwright_crawler._playwright_crawler] INFO  Final request statistics:

As you can see, the User-Agent value hasn't changed.

My goal is to replace the default Playwright user agent with my custom one, and I want to achieve this manually in my code, not automatically by crawlee-python.

Note: I tested the same with HttpCrawler, and it correctly passes the User-Agent value.

[Mantisus]

You didn't miss anything, I completely missed that in Playwright the page.set_extra_http_headers method doesn't overwrite those headers that are set at the context level.

And since User-Agent is set at the context level from HeaderGenerator we can't overwrite it from Request

When the fix for #755 is ready, that should allow you to set User-Agent for context.

But we need to think about cases where some header can be overridden at the page level from Request