airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Baron Penman <richard.pen...@gmail.com>
Subject Re: Terminate task process through UI
Date Mon, 18 Sep 2017 20:51:15 GMT
Thanks Joy - that solution works for me.

However I found then that the state of the instance is lost -
apparently the operator constructor is called again.
For example the BashOperator stores a reference to the process in
self.sp: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/bash_operator.py#L89
And then calling on_kill will raise an AttributeError - is this as expected?

On Wed, Sep 13, 2017 at 2:50 PM, Joy Gao <joyg@wepay.com> wrote:
> Hi Richard,
>
> Can confirm that the operator's on_kill() is not triggered via `clear`,
> only job.on_kill() is triggered, but it doesn't propagate to the
> operator/task. From what I can tell, currently the only way to trigger
> on_kill() is via manually sending a SIGTERM to the process while the task
> is running (code ref
> <https://github.com/apache/incubator-airflow/blob/a8d486de337779e6319b90215fd8ea7faa5f5913/airflow/models.py#L1436-L1441>),
> or if the task actually times out (code ref
> <https://github.com/apache/incubator-airflow/blob/a8d486de337779e6319b90215fd8ea7faa5f5913/airflow/models.py#L1458>
> ).
>
> I assume that the intended behavior is to trigger operator's on_kill()
> method whenever a task is cleared. If so, I'd propose patching
> theLocalTaskJob's on_kill() method as below in a PR:
>
>     def on_kill(self):
> *        self.task_instance.task.on_kill()*
>         self.task_runner.terminate()
>         self.task_runner.on_finish()
>
> Hope this helps!
>
> On Wed, Sep 13, 2017 at 8:37 AM, Richard Baron Penman <
> richard.penman@gmail.com> wrote:
>
>> Thanks Ash.
>> In my logs I see "Job shut down externally", so that BaseJob.kill()
>> method is certainly executed.
>>
>> My understanding is this then calls self.on_kill():
>> https://github.com/apache/incubator-airflow/blob/
>> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/jobs.py#L2512
>>
>> which terminates the process:
>> https://github.com/apache/incubator-airflow/blob/
>> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/task_
>> runner/bash_task_runner.py#L34
>>
>> How is the operators on_kill() method then triggered? According to my
>> logging it is not executed.
>>
>>
>> On Fri, Sep 8, 2017 at 4:49 AM, Ash Berlin-Taylor
>> <ash_airflowlist@firemirror.com> wrote:
>> > That sets the task instance state to SHUTDOWN, which is then picked up
>> by the task sentinel here https://github.com/apache/
>> incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce
>> 69623c95ce/airflow/jobs.py#L164-L165 <https://github.com/apache/
>> incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce
>> 69623c95ce/airflow/jobs.py#L164-L165> which triggers kill, so it is meant
>> to work.
>> >
>> > -ash
>> >
>> >> On 7 Sep 2017, at 20:00, Richard Baron Penman <richard.penman@gmail.com>
>> wrote:
>> >>
>> >> Is this known to work?
>> >> I tried clearing and the on_kill() method wasn't triggered.
>> >>
>> >> As far as I can tell the clearing is performed by the
>> >> clear_task_instances() function, which changes the task status but does
>> not
>> >> kill the process:
>> >> https://github.com/apache/incubator-airflow/blob/
>> b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/models.py#L120
>> >>
>> >>
>> >>
>> >> On Thu, Sep 7, 2017 at 4:46 AM, Maxime Beauchemin <
>> >> maximebeauchemin@gmail.com> wrote:
>> >>
>> >>> "Clearing" a running task instance either through the UI or through
>> the CLI
>> >>> should trigger on_kill.
>> >>>
>> >>> Max
>> >>>
>> >>> On Wed, Sep 6, 2017 at 11:45 AM, Richard Baron Penman <
>> richardbp@gmail.com
>> >>>>
>> >>> wrote:
>> >>>
>> >>>> Hello,
>> >>>>
>> >>>> Is there a way to trigger the on_kill() method of an operator through
>> the
>> >>>> UI?
>> >>>> I tried changing the state of a running task and deleting it, but
>> neither
>> >>>> approach triggered the on_kill() method and so the process was still
>> >>>> running.
>> >>>>
>> >>>> Similar question was asked on SO:
>> >>>> https://stackoverflow.com/questions/43626844/kill-the-
>> >>>> airflow-task-running-on-a-remote-location-through-airflow-ui
>> >>>>
>> >>>> Richard
>> >>>>
>> >>>
>> >
>>
>>
>
>
> --
>
> Joy Gao
> Software Engineer
> 350 Convention Way, Suite 200
> Redwood City, CA 94063
> Mobile:  669-224-9305
>
> Payments partner to the platform economy

Mime
View raw message