airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matus Valo <Matus.V...@ness.com>
Subject scheduler running on multiple nodes
Date Thu, 09 Feb 2017 08:37:23 GMT
Hey all,

We are considering deployment of airflow as our new pipeline framework. I have found out multiple
articles explaining deployment of airflow in distributed environment (e.g. [1]). Unfortunately,
I was not able to find out any use case where scheduler is deployed distributed on multiple
nodes. Is it possible to have scheduler distributed on multiple nodes to prevent single point
of failure? I haven't found any mention about it in documentation. I have found out in [2]
that it is not possible but on the other hand in [3] is reference that this can be solved
in new version of airflow.

Thanks,

[1] http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/
[2] https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME
[3] https://issues.apache.org/jira/browse/AIRFLOW-678

Matúš Valo - Senior Software Engineer
Ness Digital Engineering
NESS KE, s.r.o., Moldavská cesta 10/B, 040 11 Košice, Slovak Republic
O: +421 55 326 24 10
matus.valo@ness.com<mailto:matus.valo@ness.com>
nesskdc.sk<http://www.nesskdc.sk/> | ness.com<http://www.ness.com/> | LinkedIn<https://www.linkedin.com/company/3050495>
| Twitter<https://twitter.com/ness_tech> | Facebook<http://www.facebook.com/nesskdc/>

____________________________
Powered By Office365

The information contained in this communication is intended solely for the use of the individual
or entity to whom it is addressed and others authorized to receive it.
It may contain confidential or legally privileged information.
If you are not the intended recipient you are hereby notified that any disclosure, copying,
distribution or taking any action in reliance on the contents of this information is strictly
prohibited and may be unlawful.
If you have received this communication in error, please notify us immediately by forwarding
this email to MailAdmin@ness.com<mailto:MailAdmin@ness.com> and then delete it from
your system.
Ness technologies is neither liable for the proper and complete transmission of the information
contained in this communication nor for any delay in its receipt.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message