Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Inconsistent header override with Route.continue_ #1686

Open
elacuesta opened this issue Dec 17, 2022 · 0 comments
Open

[BUG] Inconsistent header override with Route.continue_ #1686

elacuesta opened this issue Dec 17, 2022 · 0 comments
Assignees
@mxschmitt
Labels
P3-collecting-feedback

Comments

@elacuesta
Copy link

Context:

  • Playwright Version: 1.28.0
  • Operating System: Linux, macOS
  • Python version: 3.9
  • Browser: Chromium, Firefox, WebKit

Code Snippet

import asyncio
from functools import partial
from playwright.async_api import async_playwright

async def handle_route(route, request, browser_name):
    headers = await request.all_headers()
    print("headers:", headers)
    override_headers = {"cookie": None}
    if browser_name == "firefox":
        override_headers["host"] = "httpbin.org"  # otherwise firefox times out
    await route.continue_(headers=override_headers)

async def run(browser_type):
    print("*" * 100)
    print("browser_type:", browser_type.name)
    browser = await browser_type.launch()
    context = await browser.new_context(
        storage_state={
            "cookies": [{"name": "foo", "value": "bar", "domain": "httpbin.org", "path": "/"}]
        }
    )
    page = await context.new_page()
    await page.route("**", partial(handle_route, browser_name=browser_type.name))
    await page.goto("https://httpbin.org/headers")
    content = await page.locator("pre").text_content()
    print("content:", content)
    await browser.close()

async def main():
    async with async_playwright() as playwright:
        await run(playwright.chromium)
        await run(playwright.webkit)
        await run(playwright.firefox)

if __name__ == "__main__":
    asyncio.run(main())

Describe the bug

Cookies

I can't seem to be able to consistently override cookies in requests using Route.continue_. I'm setting foo=bar as cookie when creating a context, and I'd like to avoid sending it afterwards. What I've tried:

  1. override_headers = {"cookie": None}. I'd expect the cookie header not to be sent, however it is sent by chromium & webkit, not by firefox.
  2. override_headers = {"cookie": "asdf=qwerty"}. I'd expect the cookie header to be "asdf=qwerty", and it is for webkit & firefox, however chromium still sends "foo=bar"
  3. override_headers = {}. Chromium & webkit send "foo=bar", firefox sends no cookie. I have no strong argument for either behavior, the problem for me is the inconsistency.

User agent

Additionally, the user agent header is handled differently by three browsers:

  • Firefox never sends it (I suspect it's because it's never present in the headers dict)
  • Webkit always sends a different value from the one in the request headers
  • Chromium sends the value from the request headers

Sorry for the long snippet ): Please let me know if there's something I'm missing or doing wrong, and thanks in advance for your time.

PS: I'm adding the full outputs below for completeness.

`override_headers = {"cookie": None}` 
****************************************************************************************************
browser_type: chromium
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'cookie': 'foo=bar', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36'}
override_headers: {'cookie': None}
content: {
  "headers": {
    "Accept-Encoding": "gzip, deflate, br", 
    "Cache-Control": "no-cache", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "Pragma": "no-cache", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-639d1c70-53698acf1ce28b894044c520"
  }
}

****************************************************************************************************
browser_type: webkit
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'}
override_headers: {'cookie': None}
content: {
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "User-Agent": "com.apple.WebKit.Networking/615.1.10+ CFNetwork/1335.0.3 Darwin/21.6.0", 
    "X-Amzn-Trace-Id": "Root=1-639d1c71-563620c8772dfef915f7245c"
  }
}

****************************************************************************************************
browser_type: firefox
headers: {'host': 'httpbin.org', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', 'upgrade-insecure-requests': '1', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'cross-site', 'connection': 'keep-alive', 'cookie': 'foo=bar'}
override_headers: {'cookie': None, 'host': 'httpbin.org'}
content: {
  "headers": {
    "Host": "httpbin.org", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "cross-site", 
    "Upgrade-Insecure-Requests": "1", 
    "X-Amzn-Trace-Id": "Root=1-639d1c74-6e8f740e685efeb55a0ef7d0"
  }
}
`override_headers = {"cookie": "asdf=qwerty"}` 
****************************************************************************************************
browser_type: chromium
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'cookie': 'foo=bar', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36'}
override_headers: {'cookie': 'asdf=qwerty'}
content: {
  "headers": {
    "Accept-Encoding": "gzip, deflate, br", 
    "Cache-Control": "no-cache", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "Pragma": "no-cache", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-639d1c8e-466d4e5611e850b919b41e94"
  }
}

****************************************************************************************************
browser_type: webkit
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'}
override_headers: {'cookie': 'asdf=qwerty'}
content: {
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
    "Cookie": "asdf=qwerty", 
    "Host": "httpbin.org", 
    "User-Agent": "com.apple.WebKit.Networking/615.1.10+ CFNetwork/1335.0.3 Darwin/21.6.0", 
    "X-Amzn-Trace-Id": "Root=1-639d1c90-3405577f7629b0267e6820b7"
  }
}

****************************************************************************************************
browser_type: firefox
headers: {'host': 'httpbin.org', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', 'upgrade-insecure-requests': '1', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'cross-site', 'connection': 'keep-alive', 'cookie': 'foo=bar'}
override_headers: {'cookie': 'asdf=qwerty', 'host': 'httpbin.org'}
content: {
  "headers": {
    "Cookie": "asdf=qwerty", 
    "Host": "httpbin.org", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "cross-site", 
    "Upgrade-Insecure-Requests": "1", 
    "X-Amzn-Trace-Id": "Root=1-639d1c92-52afd6571cd078e06efd2f53"
  }
}
`override_headers = {}` 
****************************************************************************************************
browser_type: chromium
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'cookie': 'foo=bar', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36'}
override_headers: {}
content: {
  "headers": {
    "Accept-Encoding": "gzip, deflate, br", 
    "Cache-Control": "no-cache", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "Pragma": "no-cache", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-639d1cbf-3f86b24407e59bbd3c515b00"
  }
}

****************************************************************************************************
browser_type: webkit
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'}
override_headers: {}
content: {
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "User-Agent": "com.apple.WebKit.Networking/615.1.10+ CFNetwork/1335.0.3 Darwin/21.6.0", 
    "X-Amzn-Trace-Id": "Root=1-639d1cc0-2dd02a8453255d434e1ae214"
  }
}

****************************************************************************************************
browser_type: firefox
headers: {'host': 'httpbin.org', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', 'upgrade-insecure-requests': '1', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'cross-site', 'connection': 'keep-alive', 'cookie': 'foo=bar'}
override_headers: {'host': 'httpbin.org'}
content: {
  "headers": {
    "Host": "httpbin.org", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "cross-site", 
    "Upgrade-Insecure-Requests": "1", 
    "X-Amzn-Trace-Id": "Root=1-639d1cc2-45d485ac48c417f652180a8c"
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mxschmitt mxschmitt

Labels
P3-collecting-feedback
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@elacuesta @rwoll @mxschmitt