airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joy Gao <j...@wepay.com>
Subject Re: Terminate task process through UI
Date Wed, 13 Sep 2017 18:50:30 GMT
Hi Richard,

Can confirm that the operator's on_kill() is not triggered via `clear`,
only job.on_kill() is triggered, but it doesn't propagate to the
operator/task. From what I can tell, currently the only way to trigger
on_kill() is via manually sending a SIGTERM to the process while the task
is running (code ref
<https://github.com/apache/incubator-airflow/blob/a8d486de337779e6319b90215fd8ea7faa5f5913/airflow/models.py#L1436-L1441>),
or if the task actually times out (code ref
<https://github.com/apache/incubator-airflow/blob/a8d486de337779e6319b90215fd8ea7faa5f5913/airflow/models.py#L1458>
).

I assume that the intended behavior is to trigger operator's on_kill()
method whenever a task is cleared. If so, I'd propose patching
theLocalTaskJob's on_kill() method as below in a PR:

    def on_kill(self):
*        self.task_instance.task.on_kill()*
        self.task_runner.terminate()
        self.task_runner.on_finish()

Hope this helps!

On Wed, Sep 13, 2017 at 8:37 AM, Richard Baron Penman <
richard.penman@gmail.com> wrote:

> Thanks Ash.
> In my logs I see "Job shut down externally", so that BaseJob.kill()
> method is certainly executed.
>
> My understanding is this then calls self.on_kill():
> https://github.com/apache/incubator-airflow/blob/
> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/jobs.py#L2512
>
> which terminates the process:
> https://github.com/apache/incubator-airflow/blob/
> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/task_
> runner/bash_task_runner.py#L34
>
> How is the operators on_kill() method then triggered? According to my
> logging it is not executed.
>
>
> On Fri, Sep 8, 2017 at 4:49 AM, Ash Berlin-Taylor
> <ash_airflowlist@firemirror.com> wrote:
> > That sets the task instance state to SHUTDOWN, which is then picked up
> by the task sentinel here https://github.com/apache/
> incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce
> 69623c95ce/airflow/jobs.py#L164-L165 <https://github.com/apache/
> incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce
> 69623c95ce/airflow/jobs.py#L164-L165> which triggers kill, so it is meant
> to work.
> >
> > -ash
> >
> >> On 7 Sep 2017, at 20:00, Richard Baron Penman <richard.penman@gmail.com>
> wrote:
> >>
> >> Is this known to work?
> >> I tried clearing and the on_kill() method wasn't triggered.
> >>
> >> As far as I can tell the clearing is performed by the
> >> clear_task_instances() function, which changes the task status but does
> not
> >> kill the process:
> >> https://github.com/apache/incubator-airflow/blob/
> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/models.py#L120
> >>
> >>
> >>
> >> On Thu, Sep 7, 2017 at 4:46 AM, Maxime Beauchemin <
> >> maximebeauchemin@gmail.com> wrote:
> >>
> >>> "Clearing" a running task instance either through the UI or through
> the CLI
> >>> should trigger on_kill.
> >>>
> >>> Max
> >>>
> >>> On Wed, Sep 6, 2017 at 11:45 AM, Richard Baron Penman <
> richardbp@gmail.com
> >>>>
> >>> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> Is there a way to trigger the on_kill() method of an operator through
> the
> >>>> UI?
> >>>> I tried changing the state of a running task and deleting it, but
> neither
> >>>> approach triggered the on_kill() method and so the process was still
> >>>> running.
> >>>>
> >>>> Similar question was asked on SO:
> >>>> https://stackoverflow.com/questions/43626844/kill-the-
> >>>> airflow-task-running-on-a-remote-location-through-airflow-ui
> >>>>
> >>>> Richard
> >>>>
> >>>
> >
>
>


-- 

Joy Gao
Software Engineer
350 Convention Way, Suite 200
Redwood City, CA 94063
Mobile:  669-224-9305

Payments partner to the platform economy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message