airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arthur Purvis <apur...@lumoslabs.com>
Subject Re: Issues with postgres_operator
Date Mon, 25 Sep 2017 15:25:20 GMT
fixed here: https://github.com/apache/incubator-airflow/pull/2513

On Mon, Sep 25, 2017 at 10:28 AM, Driesprong, Fokko <fokko@driesprong.frl>
wrote:

> Hi Shoumitra,
>
> So if I understand correctly, the query finishes but the connection does
> not close. What is the result of the query? Maybe you can also share some
> code of how you use the PostgresOperator the communicate with Redshift.
>
> Regarding your config. The sequential executor is only recommended when
> debugging, since it runs in process and only allows you to execute one task
> at the time. I would recommend upgrading to the LocalExecutor and use an
> AWS managed Postgres for the database.
>
> Let me know.
>
> Cheers, Fokko
>
> 2017-09-25 7:44 GMT+02:00 Shoumitra Srivastava <shoumitra362@gmail.com>:
>
> > Hey guys,
> >
> > I am currently using Airflow 1.8.2 to schedule some EMR tasks and then
> > execute some long running queries on our Redshift cluster. For that
> purpose
> > I am using the postgres_operator. The queries take about 30 minutes to
> run.
> > However, once they are done, the connection never closes and the operator
> > runs for an hour and a half more till its terminated at the 2 hour mark
> > every time. The logs say that the server closed the connection
> > unexpectedly. I've checked the logs on Redshift's end and it shows the
> > queries have run and the connection has been closed. Somehow, that is
> never
> > communicated back to Airflow. Any directions on what more I could check?
> To
> > give some more info, my Airflow installation is an extension of the
> > https://github.com/puckel/docker-airflow docker image, is run in an ECS
> > cluster and has SQLite as backend since I am still testing Airflow out. I
> > am also using Sequential Executor right now. I would appreciate any help
> in
> > this matter.
> >
> > thanks,
> > Shoumitra
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message