airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bolke de Bruin <bdbr...@gmail.com>
Subject Re: Airflow 1.9.0 status
Date Thu, 28 Sep 2017 20:09:29 GMT
Hi Chris

Can I suggest releasing a beta? The stable branch is only cut at RC time. Betas allow us a
broader exposure. It also gives us a point of reference.

In addition the list below are mostly longer standing issues that are also part of the 1.8.x
branch. Maybe only consider 1611, 1525, 1258, and 976 as blocker?

Cheers
Bolke

Verstuurd vanaf mijn iPad

> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <criccomini@apache.org> het volgende
geschreven:
> 
> Hey all,
> 
> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
> seeing as there are several outstanding bugs, I'm going to delay. Here are
> the bugs that I'm tracking:
> 
> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked as
> AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception for
> @on
> AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
> AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for @once
> AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is Not
> be
> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
> 
> These are the priority issues. Once they're merged, I'll cut the
> v1-9-stable and beta release.
> 
> If you can help clean this up, that would be really appreciated.
> 
> Cheers,
> Chris
> 
> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <criccomini@apache.org>
> wrote:
> 
>> Marked it for 1.9.0.
>> 
>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cjones@simpli.fi> wrote:
>>> 
>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA callbacks
>>> are not working correctly without emails... Its not a major bug, but it
>>> does cause us some annoyance in our current deployment.
>>> 
>>> Link to Jira:
>>> https://issues.apache.org/jira/browse/AIRFLOW-988
>>> 
>>> Link to PR:
>>> https://github.com/apache/incubator-airflow/pull/2415
>>> 
>>> Thanks!
>>> Charlie Jones
>>> 
>>> CHARLIE JONES
>>> Data Engineer
>>> cjones@simpli.fi  |  M: 972.821.7631
>>> __________________________________________________
>>> 
>>> 
>>> Programmatic Performance.* Localized.*
>>> __________________________________________________
>>> 
>>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>>> 800.840.0768  |  www.simpli.fi
>>> 
>>> 
>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <criccomini@apache.org>
>>> wrote:
>>> 
>>>> Merged.
>>>> 
>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>>> ryan.buckley@bluecore.com>
>>>> wrote:
>>>> 
>>>>> Would it be possible to include AIRFLOW-1587?
>>>>> Running dags from the UI is currently broken on the 1.9.0 branch due
>>> to
>>>>> this issue.
>>>>> 
>>>>> https://github.com/apache/incubator-airflow/pull/2590
>>>>> 
>>>>> Thanks,
>>>>> Ryan
>>>>> 
>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>>> <fokko@driesprong.frl
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>>>>>> https://github.com/apache/incubator-airflow/pull/2631
>>>>>> 
>>>>>> Currently importing a custom logging configuration is not work (as
>>> far
>>>>> as I
>>>>>> know). Any feedback on the PR would also be appreciated.
>>>>>> 
>>>>>> Cheers, Fokko
>>>>>> 
>>>>>> 
>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <criccomini@apache.org>:
>>>>>> 
>>>>>>> Done!
>>>>>>> 
>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>>>>>>> michael.crawford@modernizingmedicine.com> wrote:
>>>>>>> 
>>>>>>>> Can you slide the aws and emr connection type fix in?
>>>>>>>> 
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 <
>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 <
>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
>>>>>>>> 
>>>>>>>> It keeps the connection type from getting blanked out on
edit
>>> for
>>>>> these
>>>>>>>> types.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini <
>>>>> criccomini@apache.org>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward
to
>>>> these
>>>>>>> fixes!
>>>>>>>>> 
>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel <
>>>>> alex.guziel@airbnb.com
>>>>>> .
>>>>>>>> invalid
>>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Can we get this in?
>>>>>>>>>> 
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>>>>>>>>>> 
>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>>>>>>>>>> 
>>>>>>>>>> It speeds up loading times a lot, so it's a good
thing to
>>> have
>>>> in
>>>>>> 1.9.
>>>>>>>>>> 
>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini
<
>>>>>>>> criccomini@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week,
then.
>>> Initial
>>>>>>> warning
>>>>>>>>>>> stands, that I will start locking down what can
get into
>>> 1.9.0
>>>> at
>>>>>>> that
>>>>>>>>>>> point.
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin
<
>>>>>> bdbruin@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> No vote indeed, just to gather feedback on
a particular
>>> fixed
>>>>>> point
>>>>>>> in
>>>>>>>>>>>> time. It also gives a bit more trust to a
tarball than to a
>>>> git
>>>>>>> pull.
>>>>>>>>>>>> 
>>>>>>>>>>>> Bolke
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini
<
>>>>> criccomini@apache.org
>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I can do a beta. Is the process significantly
different?
>>>> IIRC,
>>>>>> it's
>>>>>>>>>>>>> basically the same, just no vote, right?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke
de Bruin <
>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Are you sure you want to go ahead
and do RCs right away?
>>>>> Isn’t a
>>>>>>>>>> beta
>>>>>>>>>>> a
>>>>>>>>>>>>>> bit smarter?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> - Bolke
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris
Riccomini <
>>>>>> criccomini@apache.org
>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I want to send out a warning
that I'm planning to cut
>>> the
>>>>>> stable
>>>>>>>>>>> branch
>>>>>>>>>>>>>>> next week, and begin the RC1
release vote. Once the
>>> stable
>>>>>> branch
>>>>>>>>>> is
>>>>>>>>>>>>>> cut, I
>>>>>>>>>>>>>>> will be locking down what commits
get cherry picked into
>>>> the
>>>>>>>>>> branch,
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> will only be doing PRs that are
required to get the
>>> release
>>>>>> out.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19
AM, Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> An update on the 1.9.0 release.
Here are the
>>> outstanding
>>>> PRs
>>>>>>> that
>>>>>>>>>>> are
>>>>>>>>>>>>>>>> slated to be included into
1.9.0:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ISSUE ID     |STATUS    |DESCRIPTION
>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open      |XSS
Vulnerability in Variable
>>>>>> endpoint
>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open      |Customize
logging in Airflow
>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened  |Fix
log source of local
>>> loggers
>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open      |Rename
the logger to log
>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open      |Fix
minor LICENSE & NOTICE
>>> issue
>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate
duplicate and
>>> unneeded
>>>>> code
>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open      |HDFSOperator
to operate HDFS
>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open      |airflow/jobs.py:create_dag_ru
>>> n()
>>>>>>>>>> exception
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> @on
>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open      |active_dagruns
shouldn't
>>> include
>>>>>> paused
>>>>>>>>>>> DAGs
>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open      |Scheduler
DAG processes can
>>> not
>>>> log
>>>>>> to
>>>>>>>>>>> stdout
>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open      |TreeView
displayed over task
>>>>>> instances
>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open      |airflow/jobs.py:manage_slas()
>>>>>> exception
>>>>>>>>>> for
>>>>>>>>>>>>>>>> @once
>>>>>>>>>>>>>>>> AIRFLOW-976  |Open      |Mark
success running task
>>> causes
>>>> it
>>>>>> to
>>>>>>>>>> fail
>>>>>>>>>>>>>>>> AIRFLOW-914  |Open      |Refactor
>>>>>> BackfillJobTest.test_backfill_
>>>>>>>>>>>>>> examples
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> AIRFLOW-913  |Open      |Refactor
>>>>>> tests.CoreTest.test_scheduler_
>>>>>>>>>> job
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>>> AIRFLOW-912  |Open      |Refactor
tests and build
>>> matrix
>>>>>>>>>>>>>>>> AIRFLOW-888  |Open      |Operators
should not push
>>> XComs
>>>> by
>>>>>>>>>> default
>>>>>>>>>>>>>>>> AIRFLOW-828  |Open      |Add
maximum size for XComs
>>>>>>>>>>>>>>>> AIRFLOW-825  |Open      |Add
Dataflow semantics
>>>>>>>>>>>>>>>> AIRFLOW-788  |Open      |Context
unexpectedly added to
>>>> hive
>>>>>> conf
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I will be locking down what
can get cherry-picked into
>>> the
>>>>>> 1.9.0
>>>>>>>>>>>> branch
>>>>>>>>>>>>>>>> shortly, so if you have something
you want in, please
>>> set
>>>>> the
>>>>>>> fix
>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>> to 1.9.0.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> We (at WePay) have deployed
1.9.0 into our dev cluster,
>>>> and
>>>>> it
>>>>>>> has
>>>>>>>>>>>> been
>>>>>>>>>>>>>>>> running smoothly for several
days.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ** I could really use help
verifying stability. If you
>>> run
>>>>>>>>>> Airflow,
>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> in your best interest to
deploy the 1.9.0 test branch
>>>>>> somewhere,
>>>>>>>>>> and
>>>>>>>>>>>>>> verify
>>>>>>>>>>>>>>>> it's working for your workload.
**
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 

Mime
View raw message