airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deng Xiaodong <>
Subject Best Practice of Airflow Setting-Up & Usage
Date Wed, 05 Sep 2018 12:56:38 GMT
Hi folks,

May you kindly share how your organization is setting up Airflow and using
it? Especially in terms of architecture. For example,

- *Setting-Up*: Do you install Airflow in a "one-time" fashion, or
containerization fashion?
- *Executor:* Which executor are you using (*LocalExecutor*,
*CeleryExecutor*, etc)? I believe most production environments are using
- *Scale*: If using Celery, normally how many worker nodes do you add? (for
sure this is up to workloads and performance of your worker nodes).
- *Queue*: if Queue feature
<> is used in your
architecture? For what advantage? (for example, explicitly assign
network-bound tasks to a worker node whose parallelism can be much higher
than its # of cores)
- *SLA*: do you have any SLA for your scheduling? (this is inspired by
@yrqls21's PR 3830 <>)
- etc.

Airflow's setting-up can be quite flexible, but I believe there is some
sort of best practice, especially in the organisations where scalability is

Thanks for sharing in advance!

Best regards,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message