Skip to content

Long-term support concerns with network interception due to Selenium Wire deprecation #3246

Closed

Description

@pradeepvegesna

We have several use cases that require request and response interception for UI testing, and we’re considering using SeleniumBase for this purpose. However, I noticed that SeleniumBase's wire mode relies on Selenium Wire, which is deprecated since January 2024.

Although network interception is still functional through Selenium Wire, I’m concerned about the impact on long-term support. Could you provide guidance on the following:

Are there plans to update or replace Selenium Wire within SeleniumBase to ensure continued support for network interception?
Is it advisable to continue using the current setup, or would you recommend alternative approaches for network interception within SeleniumBase?
Any insights on future plans for handling network intercepts would be greatly appreciated.

Activity

  1. added
    questionSomeone is looking for answers
    workaround existsYou can reach your destination if you do this...
    on Nov 6, 2024
  2. mdmintz commented on Nov 6, 2024

    @mdmintz
    Member

    The Selenium-Wire replacement is already included within SeleniumBase's new CDP Mode.

    Here's an example of that, (SeleniumBase/examples/cdp_mode/raw_res_sb.py):

    """Using CDP.network.ResponseReceived and CDP.network.RequestWillBeSent."""
    import colorama
    import mycdp
    import sys
    from seleniumbase import SB
    
    c1 = colorama.Fore.BLUE + colorama.Back.LIGHTYELLOW_EX
    c2 = colorama.Fore.BLUE + colorama.Back.LIGHTGREEN_EX
    cr = colorama.Style.RESET_ALL
    if "linux" in sys.platform:
        c1 = c2 = cr = ""
    
    async def send_handler(event: mycdp.network.RequestWillBeSent):
        r = event.request
        s = f"{r.method} {r.url}"
        for k, v in r.headers.items():
            s += f"\n\t{k} : {v}"
        print(c1 + "*** ==> RequestWillBeSent <== ***" + cr)
        print(s)
    
    async def receive_handler(event: mycdp.network.ResponseReceived):
        print(c2 + "*** ==> ResponseReceived <== ***" + cr)
        print(event.response)
    
    with SB(uc=True, test=True, locale_code="en") as sb:
        sb.activate_cdp_mode("about:blank")
        sb.cdp.add_handler(mycdp.network.RequestWillBeSent, send_handler)
        sb.cdp.add_handler(mycdp.network.ResponseReceived, receive_handler)
        url = "https://seleniumbase.io/apps/calculator"
        sb.cdp.open(url)
        sb.sleep(1)

    Usage is different from regular Selenium-Wire, but it can do all the same things (and more) with better flexibility/control.

    The greatest benefit for some people is that CDP Mode is part of UC Mode, which provides stealth to avoid bot-detection.
    The old Selenium-Wire integration is not compatible with UC Mode, but CDP Mode patches the previous issues with Selenium-Wire that prevented compatibility.

    Here's another example, (SeleniumBase/examples/cdp_mode/raw_req_sb.py), where specific requests were filtered out (intercepted and blocked), to prevent images from loading:

    """Using CDP.fetch.RequestPaused to filter content in real-time."""
    import mycdp
    from seleniumbase import SB
    
    async def request_paused_handler(event, tab):
        r = event.request
        is_image = ".png" in r.url or ".jpg" in r.url or ".gif" in r.url
        if not is_image:  # Let the data through
            tab.feed_cdp(mycdp.fetch.continue_request(request_id=event.request_id))
        else:  # Block the data (images)
            TIMED_OUT = mycdp.network.ErrorReason.TIMED_OUT
            s = f"BLOCKING | {r.method} | {r.url}"
            print(f" >>> ------------\n{s}")
            tab.feed_cdp(mycdp.fetch.fail_request(event.request_id, TIMED_OUT))
    
    with SB(uc=True, test=True, locale_code="en") as sb:
        sb.activate_cdp_mode("about:blank")
        sb.cdp.add_handler(mycdp.fetch.RequestPaused, request_paused_handler)
        url = "https://gettyimages.com/photos/firefly-2003-nathan"
        sb.cdp.open(url)
        sb.sleep(5)

    If people don't need the stealth features or the other improvements made to intercepting and handling network requests & responses, then they can continue using the existing Selenium-Wire integration as it currently stands. Or, if people want the upgrades, then the new integration is ready in CDP Mode, as shown above.

  3. mdmintz commented on Nov 6, 2024

    @mdmintz
    Member

    Closing this in favor of #3247.

  4. unpacify commented on Dec 23, 2024

    @unpacify

    okay so i tried this out with url = "https://www.kohls.com/product/prd-6908521/girls-5-16-zeroxposur-2-pc-cozy-active-knit-top-jogger-set.jsp?color=Plum&storeIds=348&prdPV=1"

    and there's only 1 request response i need, and it doesnt return it. that's the "https://www.kohls.com/web/productInventoryPrice/6908521?storeNum=348" request, which usually returns a 403 error.

    the depreciated selenium wire works and retrieves that request response, but is causing massive amounts of memory to be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature/fix already existsUpgrade to the latest version as neededquestionSomeone is looking for answersworkaround existsYou can reach your destination if you do this...

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions