airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ricky Shi <xiao.x....@gmail.com>
Subject Possible bug: Airflow frequently fail with AWS RDS backend when #tasks increases
Date Sun, 16 Aug 2020 00:03:49 GMT
Hi Everyone,

we encountered a very strange issue with airflow using AWS RDS as backend.
We found that when the number of tasks is big enough (>60), airflow will
fail with the error message (MySQL RDS backend)

sqlalchemy.exc.OperationalError:
(MySQLdb._exceptions.OperationalError) (2005, "Unknown MySQL server
host ... $AWS RDS address)

or (Postgres RDS backend):

psycopg2.OperationalError: could not translate host name $AWS RDS address


When we restart airflow, it becomes fine; and the job scheduler & website
are both running fine. However, it will fail again after a couple of days
of smooth running, with the same error message.

We found that on stack overflow, there are other ppl experiencing the same
issue but no solution found. Anyone knows how to resolve the issue?

Thanks,

-- 
Ricky Shi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message