airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Potiuk <Jarek.Pot...@polidea.com>
Subject Re: Simplifying the KubernetesExecutor
Date Thu, 13 Aug 2020 19:57:48 GMT
+1 for pod mutation in config. It's not a "yamly" thing - it's a python
code to run, so it's quite appropriate to have it in config.

On Wed, Aug 12, 2020 at 9:43 PM Daniel Imberman <daniel.imberman@gmail.com>
wrote:

> I think pod_mutation is important as an “admin override”. E.g. there might
> be a label that is required by security policy for all tasks launched by
> airflow. I do agree that the executor_config should take in a V1Pod object
>
> One option is that the executor_config could take in a pod_mutation_hook
> function as well? So users can manually override the pod functionally
> rather than just giving an object and hoping it merges the way they expect
> it to?
> via Newton Mail [
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2
> ]
> On Wed, Aug 12, 2020 at 12:19 PM, David Lum <davidlum91@gmail.com> wrote:
> I'll just leave some passing thoughts here. How the pod template file is
> modified by the executor config might not be very well defined. Currently
> when a user passes in a k8s.V1Pod object through executor_config we kind of
> arbitrarily merge the two pod definitions to what we imagine the client
> would want, but it could be often misinterpreted. Previously the
> Airflow-defined Pod class was fairly flat. This made merging the two
> objects fairly straight forward. If the user wrote the field in
> executor_config, it overwrote that field in the Pod definition, but the
> rest stayed the same. When attributes are arbitrarily nested like in the
> k8s.V1Pod, some aspects can be mistakenly overwritten. For example,
> specifying the volumeMounts of a container requires also specifying the
> rest of the V1Pod. How do we handle this? Do we just overwrite
> volumeMounts? The client must have specified a container, so do we
> overwrite that too? The fact that there is a choice means the API could be
> unintuitive to the user.
>
> What I suggest might be easier is that executor_config completely replaces
> the pod_template_file if it is specified, but then also offer something
> like pod_mutation_hook at the operator level. This makes it quite clear to
> the user, you are either replacing the pod_template, or you are modifying
> the pod_template with pod_mutation_hook and you are doing it by your own
> rules. the argument to executor_config can be something like
> ```
> executor_config: { 'KubernetesExector': {'pod_mutation': some_func,
> 'pod_template': k8s.V1Pod(...) }} # Pod template could also be a path to a
> file
> ```
>
> If both are specified then pod_mutation is a function applied to
> pod_template. Thoughts?
>
> On Wed, Aug 12, 2020 at 11:24 AM Tomasz Urbaszek <turbaszek@apache.org>
> wrote:
>
> > +1 for the idea, thanks Daniel! I agree that multi-executor and pod
> > templates should be 2.1.
> >
> > T.
> >
> > On Wed, Aug 12, 2020 at 5:13 PM Daniel Imberman <
> daniel.imberman@gmail.com>
> > wrote:
> >
> >> (Also funny enough we could use a lot of the existing infrastructure int
> >> 1.10 to create that migration script. Would just need to take the python
> >> dictionary and use the python yaml library)
> >>
> >> via Newton Mail [
> >>
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2
> >> ]
> >> On Wed, Aug 12, 2020 at 8:09 AM, Daniel Imberman <
> >> daniel.imberman@gmail.com> wrote:
> >> 100% agreed on timing. I think 2.0 should be for the breaking aspect
> >> (losing the configurations) and then 2.1/2.2 we can start adding on the
> new
> >> features. I also like the idea of a migration tool. We can make a script
> >> that takes your airflow.cfg and converts that into a yaml file.
> >>
> >> via Newton Mail [
> >>
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2
> >> ]
> >> On Wed, Aug 12, 2020 at 7:48 AM, Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> >> wrote:
> >> Big +1. All the arguments are very appealing to me and simplifying the
> >> Kubernetes Executor down to YAML-configurable one seems like a
> no-brainer
> >> especially if we provide some migration tools. I've lost countless hours
> >> on
> >> debugging some configuration problems, simply because the relevant
> >> Kubernetes-related configuration was in the least expected place - i.e.
> >> airflow.cfg. YAML configuration.
> >>
> >> I am also a big fan of both 1. and 2. I've implemented a POC of
> >> queue-based multi-scheduler once but having it embedded as part of core
> >> Airflow rather than based it on queues (which are basically a Celery
> >> Executor concept) is I think much better approach. Both 1. and 2. are
> >> cool.
> >>
> >> Now - question about timing. If we decide to go that route - my view is
> >> that simplifying Kubernetes should be an Airflow 2.0 task - alongside
> more
> >> comprehensive tests (which will be much easier to write in this case).
> The
> >> new features/ideas 1. 2. for KE I think should come after that - when we
> >> release and stabilize 2.0. Sounds like great candidates for 2.1 to me.
> >>
> >> J.
> >>
> >>
> >> On Wed, Aug 12, 2020 at 4:24 PM Daniel Imberman <
> >> daniel.imberman@gmail.com>
> >> wrote:
> >>
> >> > Hello, fellow Airflowers! I hope you are all well in these trying
> times.
> >> >
> >> >
> >> > With the recent launch of Airflow 2.0 preparation, it now seems like a
> >> > good time to review the project's state and where we can fit in some
> >> > breaking changes that will improve the project for the future.
> >> >
> >> >
> >> > When we first created the KubernetesExecutor, we had two goals in
> mind.
> >> > The first goal was to improve the airflow Auto scaling story.
> >> Previously,
> >> > airflow users would have to manually provision celery workers, which
> >> could
> >> > lead to wasted resources or missed SLAs. The other goal was to
> >> introduce a
> >> > community that was not yet well versed in the Kubernetes API to the
> >> > Kubernetes system.
> >> >
> >> >
> >> > To ease the community's transition, we abstracted many of the
> >> complexities
> >> > of creating a Kubernetes object. We chose to offer a limited number of
> >> > configurations and keep much of the pod creation process internal to
> >> > airflow. In the short-term, this system lowered the barrier to entry.
> >> Over
> >> > time, however, this abstraction has become a nightmare of tech debt as
> >> the
> >> > Kubernetes API is expensive and constantly changing.
> >> >
> >> >
> >> > With this in mind, I think it's time for us to consider a more
> >> > straightforward approach that takes the complexity out of Airflow and
> >> > offers the full Kubernetes API to the airflow user.
> >> >
> >> >
> >> > What I'm proposing here is pretty straightforward. We remove all
> >> > Kubernetes pod creation configurations from the airflow.cfg and
> instead
> >> > offer only one way to use the KubernetesExecutor: with a YAML file.
> >> >
> >> >
> >> > We can easily supply all of the configurations to the
> KubernetesExecutor
> >> > by offering example YAMLs (git sync mode is just a sidecar and an init
> >> > container, DAG volumes are just an example volume and volume mount,
> >> etc.).
> >> >
> >> >
> >> > This system would simplify a user's ability to predict what a pod will
> >> > look like once it is launched by airflow. They will know it's a base
> pod
> >> > and will be able to simply modify the pod object using the executor
> >> config
> >> > and the pod mutation hook.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > This simplification could also lead to some pretty great new features
> in
> >> > the KubernetesExecutor
> >> >
> >> >
> >> > Idea 1: Picking a pod_template_file per-task
> >> > Along with the existing customization with the executor config, solely
> >> > relying on pod files can allow users to pick the pod template file
> that
> >> > they want to use as their base pod on a per-task basis. An Airflow
> >> engineer
> >> > could supply several pre-made templates for their data scientists to
> >> reduce
> >> > the amount of customization an airflow user would need to use.
> >> >
> >> >
> >> > Idea 2: Merging the KubernetesExecutor into the CeleryExecutor
> >> > One idea that we've been excited about recently has been the idea of
> >> > creating merged Celery and Kubernetes executor. This hybrid executor
> >> would
> >> > default to launching celery workers with KEDA and would have the
> option
> >> to
> >> > launch individual tasks using the Kubernetes executor when a user
> wants
> >> > isolation or customization. Simplifying the Kubernetes executor
> reduces
> >> the
> >> > number of fail-points that this merged executor would need to account
> >> for.
> >> >
> >> >
> >> >
> >> > What would we need to do to implement this?
> >> > The good news here is that the hard work has already been done! As of
> >> > AIRFLOW-5413 [https://issues.apache.org/jira/browse/AIRFLOW-5413] by
> >> > David Lum, airflow already has the ability to use base worker pods on
> a
> >> > template file. This would involve primarily code deletion and very
> >> little
> >> > new code.
> >> >
> >> >
> >> > Thank you for your time and I look forward to the community’s
> >> discussion.
> >> >
> >> > Daniel
> >>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >
> >



-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message