airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.8.0 Release Candidate 1
Date Wed, 08 Feb 2017 06:21:50 GMT
We tracked Sid’s issue down to an issue that can happen if you use pools and do an upgrade
from an earlier release while these pools are in use. Due to the flaky behaviour of pools
in 1.7.1.3 and the tightened integrity on pools in 1.8 it can happen that a pool can have
queued tasks, but will never schedule them. The workaround is to temporarily increase the
size of the queue. This wil be noted in UPDATING.md

What is required is to have a garbage collection for pools or a verification mechanism between
scheduler and executor that pools are really in use. This should be targeted for 1.8.1.

- Bolke

> On 7 Feb 2017, at 22:24, Ali Naqvi <ali.naqvi@conornash.com> wrote:
> 
> Hi Sid,
> The deprecation warning is appearing only in the webserver, but is not
> preventing the scheduler or worker from completing any tasks.
> When you say "Most of our DAGs broke" do you mean they cannot be scheduled
> at all?
> 
> Re upgradedb
> we took a snapshot of our db on AWS and used that as a starting point for
> our upgrade. For the migration `airflow upgradedb` worked. We are using MySQL
> 5.6.27.
> 
> Cheers,
> Ali
> 
> On Tue, Feb 7, 2017 at 3:46 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
> 
>> Apache scrubs patches, so I will cherry pick it in this case. Will figure
>> out a better way to this.
>> 
>> Bolke
>> 
>>> On 7 Feb 2017, at 01:30, Dan Davydov <dan.davydov@airbnb.com.INVALID>
>> wrote:
>>> 
>>> Bolke, attached is the patch for the cgroups fix. Let me know which
>> branches you would like me to merge it to. If anyone has complaints about
>> the patch let me know (but it does not touch the core of airflow, only the
>> new cgroups task runner).
>>> 
>>> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand <sanand@apache.org
>> <mailto:sanand@apache.org>> wrote:
>>> Actually, I see the error is further down..
>>> 
>>>  File
>>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
>> line
>>> 469, in do_execute
>>> 
>>>    cursor.execute(statement, parameters)
>>> 
>>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
>>> column "dag_id" violates not-null constraint
>>> 
>>> DETAIL:  Failing row contains (null, running, 1, f).
>>> 
>>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
>>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
>>> 'dirty': False}]
>>> 
>>> It looks like an autoincrement is missing for this table.
>>> 
>>> 
>>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
>> version
>>> greater than 0.9.8
>>> 
>>> -s
>>> 
>>> 
>>> 
>>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand <sanand@apache.org
>> <mailto:sanand@apache.org>> wrote:
>>> 
>>>> I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/>
>>>> airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
>>>> upgradedb didn't quite work. First, I thought it completed
>> successfully,
>>>> then saw errors some tables were indeed missing. I ran it again and
>>>> encountered the following exception :
>>>> 
>>>> DB: postgresql://app_cousteau@db-cousteau.ep.stage.agari.com:
>> 5432/airflow <http://app_cousteau@db-cousteau.ep.stage.agari.com:
>> 5432/airflow>
>>>> 
>>>> [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
>>>> 
>>>> INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
>>>> 
>>>> INFO  [alembic.runtime.migration] Will assume transactional DDL.
>>>> 
>>>> INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
>>>> 211e584da130, add TI state index
>>>> 
>>>> INFO  [alembic.runtime.migration] Running upgrade 211e584da130 ->
>>>> 64de9cddf6c9, add task fails journal table
>>>> 
>>>> INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
>>>> f2ca10b85618, add dag_stats table
>>>> 
>>>> INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
>>>> 4addfa1236f1, Add fractional seconds to mysql tables
>>>> 
>>>> INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
>>>> 8504051e801b, xcom dag task indices
>>>> 
>>>> INFO  [alembic.runtime.migration] Running upgrade 8504051e801b ->
>>>> 5e7d17757c7a, add pid field to TaskInstance
>>>> 
>>>> INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
>>>> 127d2bf2dfa7, Add dag_id/state index on dag_run table
>>>> 
>>>> /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
>>>> SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
>> primary
>>>> key for table 'dag_stats', but has no Python-side or server-side
>> default
>>>> generator indicated, nor does it indicate 'autoincrement=True' or
>>>> 'nullable=True', and no explicit value is passed.  Primary key columns
>>>> typically may not store NULL. Note that as of SQLAlchemy 1.1,
>>>> 'autoincrement=True' must be indicated explicitly for composite (e.g.
>>>> multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is
>>>> expected for one of the columns in the primary key. CREATE TABLE
>> statements
>>>> are impacted by this change as well on most backends.
>>>> 
>>> 
>> 
>> 


Mime
View raw message