HELM CHART over ARGOCD on OpenShift -> Scheudular disconnections #26363

m3t41 · 2022-09-13T06:13:42Z

m3t41
Sep 13, 2022

Apache Airflow version

2.3.4

What happened

I am an SRE, and i have a Problem to get the Helm Chart up und Running on an OpenShift Cluster.

changes i made to the chart

Executor = KubernetesExecutor
SecurityContext disable in all Stages
Set Limits in all Stages
Redis = disable

that's it

and then i get these errors
�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efc70>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0a60>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0f40>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efe80>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808040>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0df0>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m

What you think should happen instead

`[38;5;6mairflow-scheduler �[38;5;5m05:50:17.69 �[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.69 �[0m�[1mWelcome to the Bitnami airflow-scheduler container�[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.69 �[0mSubscribe to project updates by watching �[1mhttps://github.com/bitnami/containers�[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.70 �[0mSubmit issues and feature requests at �[1mhttps://github.com/bitnami/containers/issues�[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.70 �[0m
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.75 �[0m�[38;5;2mINFO �[0m ==> Enabling non-root system user with nss_wrapper
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.76 �[0m�[38;5;2mINFO �[0m ==> ** Starting Airflow setup **
�[38;5;6mairflow-scheduler �[38;5;5m05:50:17.87 �[0m�[38;5;2mINFO �[0m ==> No injected configuration file found. Creating default config file
�[38;5;6mairflow-scheduler �[38;5;5m05:50:19.65 �[0m�[38;5;2mINFO �[0m ==> Configuring Airflow database
�[38;5;6mairflow-scheduler �[38;5;5m05:50:19.67 �[0m�[38;5;2mINFO �[0m ==> Trying to connect to the database server
�[38;5;6mairflow-scheduler �[38;5;5m05:50:23.37 �[0m�[38;5;2mINFO �[0m ==> Waiting for Airflow Webserser to be up
�[38;5;6mairflow-scheduler �[38;5;5m05:50:23.45 �[0m�[38;5;2mINFO �[0m ==> ** Airflow setup finished! **

�[38;5;6mairflow-scheduler �[38;5;5m05:50:23.47 �[0m�[38;5;2mINFO �[0m ==> ** Starting Airflow **

____ |( )_______ / /________ __
____ /| |_ /__ / / __ / __ _ | /| / /
___ ___ | / _ / _ / _ / / // / |/ |/ /
// |// // // // _/____/|__/
[�[34m2022-09-13 05:50:31,573�[0m] {�[34mscheduler_job.py:�[0m708} INFO�[0m - Starting the scheduler�[0m
[�[34m2022-09-13 05:50:31,573�[0m] {�[34mscheduler_job.py:�[0m713} INFO�[0m - Processing each file at most -1 times�[0m
[�[34m2022-09-13 05:50:31,578�[0m] {�[34mexecutor_loader.py:�[0m105} INFO�[0m - Loaded executor: KubernetesExecutor�[0m
[�[34m2022-09-13 05:50:31,675�[0m] {�[34mkubernetes_executor.py:�[0m520} INFO�[0m - Start Kubernetes executor�[0m
[�[34m2022-09-13 05:50:31,856�[0m] {�[34mkubernetes_executor.py:�[0m128} INFO�[0m - Event: and now my watch begins starting at resource_version: 0�[0m
[�[34m2022-09-13 05:50:31,875�[0m] {�[34mkubernetes_executor.py:�[0m469} INFO�[0m - Found 0 queued task instances�[0m
[�[34m2022-09-13 05:50:31,957�[0m] {�[34mmanager.py:�[0m160} INFO�[0m - Launched DagFileProcessorManager with pid: 124�[0m
[�[34m2022-09-13 05:50:31,959�[0m] {�[34mscheduler_job.py:�[0m1233} INFO�[0m - Resetting orphaned tasks for active dag runs�[0m
[�[34m2022-09-13 05:50:31,965�[0m] {�[34msettings.py:�[0m55} INFO�[0m - Configured default timezone Timezone('UTC')�[0m
[�[34m2022-09-13 05:50:31,969�[0m] {�[34mscheduler_job.py:�[0m1256} INFO�[0m - Marked 1 SchedulerJob instances as failed�[0m
[�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efc70>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:52:43,424�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0a60>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0f40>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:54:54,496�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efe80>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808040>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True�[0m
[�[34m2022-09-13 05:57:05,568�[0m] {�[34mconnectionpool.py:�[0m810} WARNING�[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0df0>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue�[0m
[�[34m2022-09-13 05:59:16,640�[0m] {�[34mkubernetes_executor.py:�[0m111} ERROR�[0m - Unknown error in KubernetesJobWatcher. Failing�[0m
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out