Skip to content

Commit

Permalink
docs: Add usage guide on Kubernetes executors (#373)
Browse files Browse the repository at this point in the history
  • Loading branch information
sbernauer authored Jan 18, 2024
1 parent 2602dff commit be8a20b
Show file tree
Hide file tree
Showing 4 changed files with 32 additions and 2 deletions.
2 changes: 1 addition & 1 deletion docs/modules/airflow/pages/usage-guide/logging.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ spec:
loggers:
"flask_appbuilder":
level: WARN
workers:
celeryExecutors:
config:
logging:
enableVectorAgent: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ spec:
max: "2"
memory:
limit: 512Mi
workers:
celeryExecutors:
config:
resources:
cpu:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
= Using Kubernetes executors

Instead of using the Celery workers you can let Airflow run the tasks using Kubernetes executors, where pods are created dynamically as needed without jobs being routed through a redis queue to the workers.

To achieve this, swap `spec.celeryExecutors` with `spec.kubernetesExecutors`.
E.g. you would change the following example

[source,yaml]
----
spec:
celeryExecutors:
roleGroups:
default:
replicas: 2
config:
resources:
# ...
----

to

[source,yaml]
----
spec:
kubernetesExecutors:
config:
resources:
# ...
----
1 change: 1 addition & 0 deletions docs/modules/airflow/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
** xref:airflow:usage-guide/security.adoc[]
** xref:airflow:usage-guide/logging.adoc[]
** xref:airflow:usage-guide/monitoring.adoc[]
** xref:airflow:usage-guide/using-kubernetes-executors.adoc[]
** xref:airflow:usage-guide/overrides.adoc[]
** xref:airflow:usage-guide/operations/index.adoc[]
*** xref:airflow:usage-guide/operations/cluster-operations.adoc[]
Expand Down

0 comments on commit be8a20b

Please sign in to comment.