How do I use selenium inside of the airflow container? #40832

haozhuang0000 · 2024-07-17T10:09:56Z

haozhuang0000
Jul 17, 2024

I try selenium-hub, but I got this error:

[2024-07-18T02:26:55.716+0000] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 363, in get
[2024-07-18T02:26:55.716+0000] {subprocess.py:93} INFO -     self.execute(Command.GET, {"url": url})
[2024-07-18T02:26:55.716+0000] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 354, in execute
[2024-07-18T02:26:55.717+0000] {subprocess.py:93} INFO -     self.error_handler.check_response(response)
[2024-07-18T02:26:55.717+0000] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
[2024-07-18T02:26:55.717+0000] {subprocess.py:93} INFO -     raise exception_class(message, screen, stacktrace)
[2024-07-18T02:26:55.717+0000] {subprocess.py:93} INFO - selenium.common.exceptions.InvalidSessionIdException: Message: Unable to find session with ID: 8f4e3b7e61a3a25ff9bda3d29b8f6834
[2024-07-18T02:26:55.718+0000] {subprocess.py:93} INFO - Build info: version: '4.20.0', revision: '866c76ca80'
[2024-07-18T02:26:55.718+0000] {subprocess.py:93} INFO - System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.146.1-microsoft-standard-WSL2', java.version: '17.0.11'
[2024-07-18T02:26:55.718+0000] {subprocess.py:93} INFO - Driver info: driver.version: unknown
[2024-07-18T02:26:55.719+0000] {subprocess.py:93} INFO - Stacktrace:
[2024-07-18T02:26:55.719+0000] {subprocess.py:93} INFO - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 8f4e3b7e61a3a25ff9bda3d29b8f6834
[2024-07-18T02:26:55.719+0000] {subprocess.py:93} INFO - Build info: version: '4.20.0', revision: '866c76ca80'
[2024-07-18T02:26:55.719+0000] {subprocess.py:93} INFO - System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.146.1-microsoft-standard-WSL2', java.version: '17.0.11'
[2024-07-18T02:26:55.720+0000] {subprocess.py:93} INFO - Driver info: driver.version: unknown
[2024-07-18T02:26:55.720+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)
[2024-07-18T02:26:55.721+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.grid.sessionmap.SessionMap.getUri(SessionMap.java:84)
[2024-07-18T02:26:55.721+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:223)
[2024-07-18T02:26:55.721+0000] {subprocess.py:93} INFO - 	at io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)
[2024-07-18T02:26:55.722+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:180)
[2024-07-18T02:26:55.722+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:383)
[2024-07-18T02:26:55.722+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
[2024-07-18T02:26:55.723+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)
[2024-07-18T02:26:55.723+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
[2024-07-18T02:26:55.723+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.grid.router.Router.execute(Router.java:87)
[2024-07-18T02:26:55.724+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)
[2024-07-18T02:26:55.724+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
[2024-07-18T02:26:55.724+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)
[2024-07-18T02:26:55.724+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
[2024-07-18T02:26:55.725+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)
[2024-07-18T02:26:55.725+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
[2024-07-18T02:26:55.725+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)
[2024-07-18T02:26:55.726+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
[2024-07-18T02:26:55.726+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)
[2024-07-18T02:26:55.726+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
[2024-07-18T02:26:55.727+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)
[2024-07-18T02:26:55.727+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Route.execute(Route.java:69)
[2024-07-18T02:26:55.727+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)
[2024-07-18T02:26:55.728+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
[2024-07-18T02:26:55.728+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
[2024-07-18T02:26:55.728+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)
[2024-07-18T02:26:55.728+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)
[2024-07-18T02:26:55.729+0000] {subprocess.py:93} INFO - 	at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)
[2024-07-18T02:26:55.729+0000] {subprocess.py:93} INFO - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[2024-07-18T02:26:55.729+0000] {subprocess.py:93} INFO - 	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
[2024-07-18T02:26:55.729+0000] {subprocess.py:93} INFO - 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[2024-07-18T02:26:55.730+0000] {subprocess.py:93} INFO - 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[2024-07-18T02:26:55.730+0000] {subprocess.py:93} INFO - 	at java.base/java.lang.Thread.run(Unknown Source)
[2024-07-18T02:26:55.730+0000] {subprocess.py:93} INFO - 
[2024-07-18T02:26:56.363+0000] {subprocess.py:97} INFO - Command exited with return code 1
[2024-07-18T02:26:56.364+0000] {taskinstance.py:441} INFO - ::group::Post task execution logs
[2024-07-18T02:26:56.374+0000] {taskinstance.py:2905} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 401, in wrapper
    return func(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/bash.py", line 243, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2024-07-18T02:26:56.377+0000] {taskinstance.py:1206} INFO - Marking task as FAILED. dag_id=10K_Production, task_id=get_data, run_id=manual__2024-07-18T02:26:48.867390+00:00, execution_date=20240718T022648, start_date=20240718T022649, end_date=20240718T022656
[2024-07-18T02:26:56.391+0000] {standard_task_runner.py:110} ERROR - Failed to execute job 1116 for task get_data (Bash command failed. The command returned a non-zero exit code 1.; 5939)
[2024-07-18T02:26:56.416+0000] {local_task_job_runner.py:243} INFO - Task exited with return code 1
[2024-07-18T02:26:56.431+0000] {taskinstance.py:3503} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2024-07-18T02:26:56.432+0000] {local_task_job_runner.py:222} INFO - ::endgroup::

I am not sure where the setup is wrong.

My docker-compose.yaml:

version: '3.7'
x-airflow-common:
  &airflow-common
  build: .
  image: ${AIRFLOW_IMAGE_NAME:-extending_airflow:latest}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: 10
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
  depends_on:
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

  selenium-hub:
    image: seleniarm/hub:latest
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
  chrome:
    image: selenium/node-chrome
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true
      - SE_NODE_MAX_SESSIONS=10
    ports:
      - 5901:5900

volumes:
  postgres-db-volume:

Python code:

Under dags:

task = BashOperator(
                        task_id='get_data',
                        bash_command=f'python {path}/dags/src/xxxxx.py'
                        dag=dag
                    )

function inside xxxxx.py that useful to call driver

def get_driver(isHeadless=False):
    options = Options()
    options.set_capability('se:recordVideo', True)
    options.set_capability('se:screenResolution', '1920x1080')
    options.set_capability('se:name', 'test_visit_basic_auth_secured_page (ChromeTests)')
    options.add_argument("--proxy-server=http://airflow-mitmproxy-1:8080")
    options.add_argument("--ignore-certificate-errors")
    options.add_argument('--headless')
    driver = webdriver.Remote('http://selenium-hub:4444', options=options)
    return driver

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How do I use selenium inside of the airflow container? #40832

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 0 comments

Select a reply

How do I use selenium inside of the airflow container? #40832

haozhuang0000 Jul 17, 2024

My docker-compose.yaml:

Python code:

Under dags:

function inside xxxxx.py that useful to call driver

Replies: 0 comments

haozhuang0000
Jul 17, 2024