airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Uz <ali...@gmail.com>
Subject Re: DAG with ExternalTaskSensor won't run after deletion
Date Tue, 12 Sep 2017 21:13:13 GMT
Hi Joy,

Indeed that is what's happening. I am running airflow (at the moment) on a
single docker instance in AWS ECS and the only way I can bounce the
scheduler is either by initiating a new deploy or by SSHing into the
machine and restarting the scheduler manually, which isn't really a
sustainable method for the long-term.
I guess I can file this one as a bug in JIRA if some other guys also
confirm that, this is indeed a bug and not an intended feature.

Thanks again!

- Ali

On Tue, Sep 12, 2017 at 10:10 PM, Joy Gao <joyg@wepay.com> wrote:

> Hi Ali,
>
> I was able to replicate your scenario above.
>
> The executor maintains a cache of running task instances. When a task
> instance finishes, it is removed from the `running` cache of the executor.
> When you hard-delete the DAG Run and the task instances via the UI, the
> corresponding task instances are still in the 'running' cache. So when the
> scheduler is iterating through the dags, it is told by the executor that
> this particular task instance is already running, and does not need to be
> rescheduled.  If you bounce the scheduler, the task should be picked up
> again as the cache will be regenerated.
>
> I believe this is a bug in the scheduler. So feel free to report it as a
> JIRA issue on this. Although in general,  I try to avoid hard-deleting
> items from the DB when they are in `running` state. Hope this helps!
>
> Joy
>
> On Mon, Sep 11, 2017 at 2:38 PM, Ali Uz <aliuz1@gmail.com> wrote:
>
> > Hello,
> >
> > I have a DAG that has an ExternalTaskSensor that waits for another DAG to
> > complete before running. Here's my scenario: I run this DAG and while the
> > external task sensor is running, I pause it, then delete all its tasks
> and
> > the DAG Run from the Admin panel. Then I turn it on again, and the DAG
> > displays as "running" in the UI, but none of its tasks get queued.
> >
> > Also in the Jobs UI there seems to be a Job stuck in "running" which
> seems
> > to be related to the DAG I just deleted.
> >
> > How can I ensure that my DAG will always run no matter what?
> >
> > Thanks,
> >
> > - Ali
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message