airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Imberman <>
Subject Simplifying the KubernetesExecutor
Date Wed, 12 Aug 2020 14:24:16 GMT
Hello, fellow Airflowers! I hope you are all well in these trying times.

With the recent launch of Airflow 2.0 preparation, it now seems like a good time to review
the project's state and where we can fit in some breaking changes that will improve the project
for the future.

When we first created the KubernetesExecutor, we had two goals in mind. The first goal was
to improve the airflow Auto scaling story. Previously, airflow users would have to manually
provision celery workers, which could lead to wasted resources or missed SLAs. The other goal
was to introduce a community that was not yet well versed in the Kubernetes API to the Kubernetes

To ease the community's transition, we abstracted many of the complexities of creating a Kubernetes
object. We chose to offer a limited number of configurations and keep much of the pod creation
process internal to airflow. In the short-term, this system lowered the barrier to entry.
Over time, however, this abstraction has become a nightmare of tech debt as the Kubernetes
API is expensive and constantly changing.

With this in mind, I think it's time for us to consider a more straightforward approach that
takes the complexity out of Airflow and offers the full Kubernetes API to the airflow user.

What I'm proposing here is pretty straightforward. We remove all Kubernetes pod creation configurations
from the airflow.cfg and instead offer only one way to use the KubernetesExecutor: with a
YAML file.

We can easily supply all of the configurations to the KubernetesExecutor by offering example
YAMLs (git sync mode is just a sidecar and an init container, DAG volumes are just an example
volume and volume mount, etc.).

This system would simplify a user's ability to predict what a pod will look like once it is
launched by airflow. They will know it's a base pod and will be able to simply modify the
pod object using the executor config and the pod mutation hook.

This simplification could also lead to some pretty great new features in the KubernetesExecutor

Idea 1: Picking a pod_template_file per-task
Along with the existing customization with the executor config, solely relying on pod files
can allow users to pick the pod template file that they want to use as their base pod on a
per-task basis. An Airflow engineer could supply several pre-made templates for their data
scientists to reduce the amount of customization an airflow user would need to use.

Idea 2: Merging the KubernetesExecutor into the CeleryExecutor
One idea that we've been excited about recently has been the idea of creating merged Celery
and Kubernetes executor. This hybrid executor would default to launching celery workers with
KEDA and would have the option to launch individual tasks using the Kubernetes executor when
a user wants isolation or customization. Simplifying the Kubernetes executor reduces the number
of fail-points that this merged executor would need to account for.

What would we need to do to implement this?
The good news here is that the hard work has already been done! As of AIRFLOW-5413 []
by David Lum, airflow already has the ability to use base worker pods on a template file.
This would involve primarily code deletion and very little new code.

Thank you for your time and I look forward to the community’s discussion.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message