Skip to content

Commit

Permalink
docs: Add usage guide on Kubernetes executors
Browse files Browse the repository at this point in the history
  • Loading branch information
sbernauer committed Jan 18, 2024
1 parent 2602dff commit 2b92945
Show file tree
Hide file tree
Showing 4 changed files with 32 additions and 2 deletions.
2 changes: 1 addition & 1 deletion docs/modules/airflow/pages/usage-guide/logging.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ spec:
loggers:
"flask_appbuilder":
level: WARN
workers:
celeryExecutors:
config:
logging:
enableVectorAgent: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ spec:
max: "2"
memory:
limit: 512Mi
workers:
celeryExecutors:
config:
resources:
cpu:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
= Using Kubernetes executors

Instead of using the Celery workers you can let Airflow run the tasks using Kubernetes executors, where pods are created dynamically as needed without jobs being routed through a redis queue to the workers.

To achieve this, swap `spec.celeryExecutors` with `spec.kubernetesExecutors`.
E.g. you would change the following example

[source,yaml]
----
spec:
celeryExecutors:
roleGroups:
default:
replicas: 2
config:
resources:
# ...
----

to

[source,yaml]
----
spec:
kubernetesExecutors:
config:
resources:
# ...
----
1 change: 1 addition & 0 deletions docs/modules/airflow/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
** xref:airflow:usage-guide/security.adoc[]
** xref:airflow:usage-guide/logging.adoc[]
** xref:airflow:usage-guide/monitoring.adoc[]
** xref:airflow:usage-guide/using-kubernetes-executors.adoc[]
** xref:airflow:usage-guide/overrides.adoc[]
** xref:airflow:usage-guide/operations/index.adoc[]
*** xref:airflow:usage-guide/operations/cluster-operations.adoc[]
Expand Down

0 comments on commit 2b92945

Please sign in to comment.