airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Guziel <alex.guz...@airbnb.com.INVALID>
Subject Re: Terminate task process through UI
Date Wed, 13 Sep 2017 21:04:52 GMT
Right now, there are a few layers of processes. Here's an example in the
celery worker case.

CeleryMainProcess -> CeleryPoolWorker -> Airflow run --local -> Airflow run
--raw -> Bash command

In the past, airflow run --raw would handle almost all logic, and --local
would just handle heartbeating, which would include checking the task
state, and sending a SIGTERM down if necessary. (--local is LocalTaskJob).
I recently moved the logic in airflow run --raw that runs before the bash
command to run in airflow run --local, but I believe the logic in Airflow
run --raw after the bash command (or whatever operator) should also be
moved up. It would be more logical in cases like these.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message