Usage within Docker - "bind() failed: Cannot assign requested address (99)"

[agentydragon]

Hi.
I have a custom Docker image that I use to test my projects. Some of my tests use these web testing rules, and I'm having trouble running them in Docker.

Here's a simple example repo with my setup: https://gitlab.com/agentydragon/webtesting-docker-test

Here's what I'd like to be able do to within that repo:

• Build the Docker container: docker build -t bazel-webtest-testrunner .
• Start the container:

  docker run -it \
    --mount type=bind,source=$(pwd),target=/repo \
    bazel-webtest-testrunner

• Within the container, run the tests successfully:

  cd /repo
  bazel test //...

The test I'm running in that repo is really simple, just loads http://python.org, tests there's "Python" in the title.

Issues I could fix

I've encountered the following issues that I've been able to fix. I'm not sure how fixable they are.

Missing urllib3

On my first try I got this backtrace:

2020/12/19 00:26:02 launching HTTP server at: 94d4d5cced37:43027
2020/12/19 00:26:02.463374 Listening on :35507
Traceback (most recent call last):
  File "execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/__main__/test.py", line 2, in <module>
    from testing.web import webtest
  File "execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/io_bazel_rules_webtesting/testing/web/webtest.py", line 27, in <module>
    from selenium.webdriver.remote import remote_connection
  File "execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/org_seleniumhq_py/selenium/webdriver/__init__.py", line 18, in <module>
    from .firefox.webdriver import WebDriver as Firefox  # noqa
  File "execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/org_seleniumhq_py/selenium/webdriver/firefox/webdriver.py", line 29, in 
<module>
    from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
  File "execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/org_seleniumhq_py/selenium/webdriver/remote/webdriver.py", line 27, in <module>
    from .remote_connection import RemoteConnection
  File "execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/org_seleniumhq_py/selenium/webdriver/remote/remote_connection.py", line 24, in <module>
    import urllib3
ModuleNotFoundError: No module named 'urllib3'

I fixed that by pip installing urllib3 in the Dockerfile.

Missing libglib, libnss3

I got first this:

io_bazel_rules_webtesting/third_party/chromedriver/chromedriver.out/chromedriver_linux64/chromedriver: error while loading shared libraries: libglib-2.0.so.0: cannot open shared object file: No such file or directory

Then this:

io_bazel_rules_webtesting/third_party/chromedriver/chromedriver.out/chromedriver_linux64/chromedriver: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

I fixed that by installing libglib2.0-0, libnss3 in the Dockerfile.

Issue I could not fix

When I run bazel test //..., I now get this log: https://pastebin.ubuntu.com/p/HcVPdKNFS4/

Here's the relevant-looking part:

2020/12/19 00:35:55 launching HTTP server at: 9eaa0e86082f:44253
2020/12/19 00:35:55.565498 Listening on :46247
Starting ChromeDriver 76.0.3809.0 (9fcb9ac87427f9327c726808c64f2e8a0c719b1a-refs/heads/master@{#664981}) on port 39807
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[1608338155.709][SEVERE]: bind() failed: Cannot assign requested address (99)
2020/12/19 00:35:58.729798 Error shutting down driver: os: process already finished
2020/12/19 00:35:58.729961 Error creating webdriver session: [Go WebDriver Client] (unknown error) unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /root/.cache/bazel/_bazel_root/6530f9eb448d96e7552a3c3a29b6cd2b/execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/io_bazel_rules_webtesting/third_party/chromium/chromium.out/chrome-linux/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Starting ChromeDriver 76.0.3809.0 (9fcb9ac87427f9327c726808c64f2e8a0c719b1a-refs/heads/master@{#664981}) on port 42569
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[1608338158.738][SEVERE]: bind() failed: Cannot assign requested address (99)
2020/12/19 00:36:01.751354 Error shutting down driver: os: process already finished
2020/12/19 00:36:01.751617 Error creating webdriver session: [Go WebDriver Client] (unknown error) unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /root/.cache/bazel/_bazel_root/6530f9eb448d96e7552a3c3a29b6cd2b/execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/io_bazel_rules_webtesting/third_party/chromium/chromium.out/chrome-linux/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Starting ChromeDriver 76.0.3809.0 (9fcb9ac87427f9327c726808c64f2e8a0c719b1a-refs/heads/master@{#664981}) on port 43375
Only local connections are allowed.
Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
[1608338161.762][SEVERE]: bind() failed: Cannot assign requested address (99)
2020/12/19 00:36:04.795563 Error shutting down driver: os: process already finished
2020/12/19 00:36:04.795834 Error creating webdriver session: [Go WebDriver Client] (unknown error) unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /root/.cache/bazel/_bazel_root/6530f9eb448d96e7552a3c3a29b6cd2b/execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/io_bazel_rules_webtesting/third_party/chromium/chromium.out/chrome-linux/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
/root/.cache/bazel/_bazel_root/6530f9eb448d96e7552a3c3a29b6cd2b/execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/org_seleniumhq_py/selenium/webdriver/remote/remote_connection.py:374: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.0.2', 58294), raddr=('172.17.0.2', 44253)>
  return self._request(command_info[0], url, body=data)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Overall, looks to me like Chromedriver crashes with bind() failed: Cannot assign requested address (99).

I've Googled for the error, and found some fixes which involved changing Chromedriver flags. Is there any easy way for me to pass additional flags to Chromedriver when using a py_web_test_suite? If there were, I could try those fixes I found.

I found this promising-looking Stack Overflow question: https://stackoverflow.com/questions/55844788/how-to-fix-severe-bind-failed-cannot-assign-requested-address-99-while

There's 2 answers. One says it's due to IPv6, which might not be enabled in my Docker setup. I have not yet tried enabling IPv6 in my Docker host. The other answer says to pass --whitelisted-ips= to Chromedriver. Is there any way to do that when Chromedriver is launched by py_web_test_suite?

Overall:

• Can I pass flags to the Chromedriver launched by py_web_test_suite? If yes, I can use that to try to resolve this on my own.
• If someone has more time to help me debug, can someone make my example repo's tests work for their setup?
• Or ideally, it would be best if there could be a working example with Docker in this repo. If I get it working, I'll try to contribute one.

[chilliams]

I was able to fix this by using a newer version of Chromium and Chromedriver.

[agentydragon]

@chilliams can you maybe share a gist / small repo where you have this working, so I can check if it also works on my usecase (which is GitLab's CI)?

I've just tried bumping my rules_webtesting dependency to 0.3.5 and to //web/versioned:browsers-0.3.3, and I'm getting the same issue.

[bduffany]

The root issue in my case was that chrome was crashing due to missing shared libraries. This line hints at the real error (the bind() failed is a red herring):

(The process started from chrome location /root/.cache/bazel/_bazel_root/6530f9eb448d96e7552a3c3a29b6cd2b/execroot/__main__/bazel-out/k8-fastbuild/bin/test_chromium-local.sh.runfiles/io_bazel_rules_webtesting/third_party/chromium/chromium.out/chrome-linux/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

If you run that binary directly:

$ /root/.cache/[...]/chrome-linux/chrome --headless

it may print out some errors like "libglib-2.0.so not found" which is the real reason Chrome is "crashing" (in fact it never starts).

Rather than attempting to install all of these libraries one-by-one (and potentially having to install more libraries in future chromium releases), I found that the easiest way to get all of the missing shared libraries was just to install chromium-browser within the Dockerfile:

# The chromium binary used by rules_webtesting is not fully hermetic and
# depends on some shared libraries being available in the environment,
# so we install chromium-browser here just to install the shared libraries that it
# depends on.
RUN apt-get update && apt-get install -y chromium-browser

This reliance on shared object files being installed on the host system means that rules_webtesting are not fully hermetic, which seems like a legitimate issue. Maybe the rules should explicitly include all of the required .so files and set LD_LIBRARY_PATH when spawning the Chrome process, or something like that (I'm not an expert in this area by any means, so this sort of approach could understandably be infeasible).

[agentydragon]

Thanks @bduffany. The "install chromium-browser" workaround unfortunately didn't work for me on its own, I suspect it might be because in newer Ubuntu it gets installed as a snap. But launching the Chrome did reveal a bunch of missing library errors.

Independently of this issue: it would be nice if we could have those errors surfaced in the test logs if they happen. If the error messages just get printed to stderr, would be nice to have an option to pipe it into the test log, so it wouldn't be this hard to get to the bottom of it.

I managed to get chrome to launch OK, but it still doesn't quite work. I'll update with progress on my testcase repo.