airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Baron Penman <richar...@gmail.com>
Subject Operators that poll vs Sensors
Date Tue, 05 Sep 2017 15:14:29 GMT
Hello,

I noticed some operators in contrib (ECS, databricks, dataproc) submit
their task and then poll until complete:
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/ecs_operator.py
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/databricks_operator.py
https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/dataproc_operator.py

Would they be better designed as Sensors?

I ask because I wrote a Sensor for an API and wondering whether there was
an advantage to the Operator polling approach.

Richard

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message