airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deng Xiaodong <>
Subject Multiple Schedulers - "scheduler_lock"
Date Fri, 01 Mar 2019 15:25:10 GMT
Hi Max,

I’m trying to prepare an AIP for supporting multiple-scheduler in Airflow (mainly for HA
and Higher scheduling performance).

Along the process of code checking, I found that there is one attribute of DagModel, “scheduler_lock”.
It’s not used at all in current implementation, but it was introduced long time back (2015)
to allow multiple schedulers to work together (

Since you were the original author of it, it would be very helpful if you can kindly share
why the multiple-schedulers implementation was removed eventually, and what challenges/complexity
there were.
(You already shared a few valuable inputs in the earlier discussion
, mainly relating to hiccups around concurrency, cross DAG prioritisation & load on DB.
Other than these, anything else you would like to advise?)

I will also dive into the git history further to understand it better.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message