airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: Airflow 1.9.0 status
Date Fri, 29 Sep 2017 22:54:26 GMT
Welp. Work got in the way, so I'll cut the beta on Monday. :)

On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <criccomini@apache.org>
wrote:

> Works for me. Will try and cut a beta before end of week.
>
> Blockers for 1.9.0 are:
>
> AIRFLOW-1611 |Bug         |Customize logging in Airflow
> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
> AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are marked
> as
> AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>
>
> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>
>> Hi Chris
>>
>> Can I suggest releasing a beta? The stable branch is only cut at RC time.
>> Betas allow us a broader exposure. It also gives us a point of reference.
>>
>> In addition the list below are mostly longer standing issues that are
>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, and
>> 976 as blocker?
>>
>> Cheers
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini <criccomini@apache.org>
>> het volgende geschreven:
>> >
>> > Hey all,
>> >
>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, but
>> > seeing as there are several outstanding bugs, I'm going to delay. Here
>> are
>> > the bugs that I'm tracking:
>> >
>> > AIRFLOW-1611 |Bug         |Customize logging in Airflow
>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue
>> > AIRFLOW-1258 |Bug         |TaskInstances within SubDagOperator are
>> marked as
>> > AIRFLOW-1055 |Bug         |airflow/jobs.py:create_dag_run() exception
>> for
>> > @on
>> > AIRFLOW-1018 |Bug         |Scheduler DAG processes can not log to stdout
>> > AIRFLOW-1013 |Bug         |airflow/jobs.py:manage_slas() exception for
>> @once
>> > AIRFLOW-988  |Bug         |SLA Miss Callbacks Are Repeated if Email is
>> Not
>> > be
>> > AIRFLOW-976  |Bug         |Mark success running task causes it to fail
>> >
>> > These are the priority issues. Once they're merged, I'll cut the
>> > v1-9-stable and beta release.
>> >
>> > If you can help clean this up, that would be really appreciated.
>> >
>> > Cheers,
>> > Chris
>> >
>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini <
>> criccomini@apache.org>
>> > wrote:
>> >
>> >> Marked it for 1.9.0.
>> >>
>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <cjones@simpli.fi>
>> wrote:
>> >>>
>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA
>> callbacks
>> >>> are not working correctly without emails... Its not a major bug, but
>> it
>> >>> does cause us some annoyance in our current deployment.
>> >>>
>> >>> Link to Jira:
>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988
>> >>>
>> >>> Link to PR:
>> >>> https://github.com/apache/incubator-airflow/pull/2415
>> >>>
>> >>> Thanks!
>> >>> Charlie Jones
>> >>>
>> >>> CHARLIE JONES
>> >>> Data Engineer
>> >>> cjones@simpli.fi  |  M: 972.821.7631
>> >>> __________________________________________________
>> >>>
>> >>>
>> >>> Programmatic Performance.* Localized.*
>> >>> __________________________________________________
>> >>>
>> >>> 1407 Texas Street  |  Suite 202  |  Fort Worth, TX 76102
>> >>> 800.840.0768  |  www.simpli.fi
>> >>>
>> >>>
>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini <
>> criccomini@apache.org>
>> >>> wrote:
>> >>>
>> >>>> Merged.
>> >>>>
>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley <
>> >>> ryan.buckley@bluecore.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Would it be possible to include AIRFLOW-1587?
>> >>>>> Running dags from the UI is currently broken on the 1.9.0 branch
due
>> >>> to
>> >>>>> this issue.
>> >>>>>
>> >>>>> https://github.com/apache/incubator-airflow/pull/2590
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Ryan
>> >>>>>
>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko
>> >>> <fokko@driesprong.frl
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi All,
>> >>>>>>
>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release:
>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631
>> >>>>>>
>> >>>>>> Currently importing a custom logging configuration is not
work (as
>> >>> far
>> >>>>> as I
>> >>>>>> know). Any feedback on the PR would also be appreciated.
>> >>>>>>
>> >>>>>> Cheers, Fokko
>> >>>>>>
>> >>>>>>
>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini <criccomini@apache.org
>> >:
>> >>>>>>
>> >>>>>>> Done!
>> >>>>>>>
>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford <
>> >>>>>>> michael.crawford@modernizingmedicine.com> wrote:
>> >>>>>>>
>> >>>>>>>> Can you slide the aws and emr connection type fix
in?
>> >>>>>>>>
>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636
<
>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636>
>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626
<
>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626>
>> >>>>>>>>
>> >>>>>>>> It keeps the connection type from getting blanked
out on edit
>> >>> for
>> >>>>> these
>> >>>>>>>> types.
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> Mike
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini
<
>> >>>>> criccomini@apache.org>
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking
forward to
>> >>>> these
>> >>>>>>> fixes!
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel
<
>> >>>>> alex.guziel@airbnb.com
>> >>>>>> .
>> >>>>>>>> invalid
>> >>>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Can we get this in?
>> >>>>>>>>>>
>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519
>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621
>> >>>>>>>>>>
>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2
>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/
>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a
>> >>>>>>>>>>
>> >>>>>>>>>> It speeds up loading times a lot, so it's
a good thing to
>> >>> have
>> >>>> in
>> >>>>>> 1.9.
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris
Riccomini <
>> >>>>>>>> criccomini@apache.org>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta
next week, then.
>> >>> Initial
>> >>>>>>> warning
>> >>>>>>>>>>> stands, that I will start locking down
what can get into
>> >>> 1.9.0
>> >>>> at
>> >>>>>>> that
>> >>>>>>>>>>> point.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke
de Bruin <
>> >>>>>> bdbruin@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> No vote indeed, just to gather feedback
on a particular
>> >>> fixed
>> >>>>>> point
>> >>>>>>> in
>> >>>>>>>>>>>> time. It also gives a bit more trust
to a tarball than to a
>> >>>> git
>> >>>>>>> pull.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Bolke
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris
Riccomini <
>> >>>>> criccomini@apache.org
>> >>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I can do a beta. Is the process
significantly different?
>> >>>> IIRC,
>> >>>>>> it's
>> >>>>>>>>>>>>> basically the same, just no
vote, right?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56
AM, Bolke de Bruin <
>> >>>>>>> bdbruin@gmail.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Are you sure you want to
go ahead and do RCs right away?
>> >>>>> Isn’t a
>> >>>>>>>>>> beta
>> >>>>>>>>>>> a
>> >>>>>>>>>>>>>> bit smarter?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> - Bolke
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41,
Chris Riccomini <
>> >>>>>> criccomini@apache.org
>> >>>>>>>>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hey all,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I want to send out a
warning that I'm planning to cut
>> >>> the
>> >>>>>> stable
>> >>>>>>>>>>> branch
>> >>>>>>>>>>>>>>> next week, and begin
the RC1 release vote. Once the
>> >>> stable
>> >>>>>> branch
>> >>>>>>>>>> is
>> >>>>>>>>>>>>>> cut, I
>> >>>>>>>>>>>>>>> will be locking down
what commits get cherry picked into
>> >>>> the
>> >>>>>>>>>> branch,
>> >>>>>>>>>>>> and
>> >>>>>>>>>>>>>>> will only be doing PRs
that are required to get the
>> >>> release
>> >>>>>> out.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>> Chris
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017
at 11:19 AM, Chris Riccomini <
>> >>>>>>>>>>>> criccomini@apache.org
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Hey all,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> An update on the
1.9.0 release. Here are the
>> >>> outstanding
>> >>>> PRs
>> >>>>>>> that
>> >>>>>>>>>>> are
>> >>>>>>>>>>>>>>>> slated to be included
into 1.9.0:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> ISSUE ID     |STATUS
   |DESCRIPTION
>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open
     |XSS Vulnerability in Variable
>> >>>>>> endpoint
>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open
     |Customize logging in Airflow
>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened
 |Fix log source of local
>> >>> loggers
>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open
     |Rename the logger to log
>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open
     |Fix minor LICENSE & NOTICE
>> >>> issue
>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In
Progres|Eliminate duplicate and
>> >>> unneeded
>> >>>>> code
>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open
     |HDFSOperator to operate HDFS
>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open
     |airflow/jobs.py:create_dag_ru
>> >>> n()
>> >>>>>>>>>> exception
>> >>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>> @on
>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open
     |active_dagruns shouldn't
>> >>> include
>> >>>>>> paused
>> >>>>>>>>>>> DAGs
>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open
     |Scheduler DAG processes can
>> >>> not
>> >>>> log
>> >>>>>> to
>> >>>>>>>>>>> stdout
>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open
     |TreeView displayed over task
>> >>>>>> instances
>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open
     |airflow/jobs.py:manage_slas()
>> >>>>>> exception
>> >>>>>>>>>> for
>> >>>>>>>>>>>>>>>> @once
>> >>>>>>>>>>>>>>>> AIRFLOW-976  |Open
     |Mark success running task
>> >>> causes
>> >>>> it
>> >>>>>> to
>> >>>>>>>>>> fail
>> >>>>>>>>>>>>>>>> AIRFLOW-914  |Open
     |Refactor
>> >>>>>> BackfillJobTest.test_backfill_
>> >>>>>>>>>>>>>> examples
>> >>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>> AIRFLOW-913  |Open
     |Refactor
>> >>>>>> tests.CoreTest.test_scheduler_
>> >>>>>>>>>> job
>> >>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>> real
>> >>>>>>>>>>>>>>>> AIRFLOW-912  |Open
     |Refactor tests and build
>> >>> matrix
>> >>>>>>>>>>>>>>>> AIRFLOW-888  |Open
     |Operators should not push
>> >>> XComs
>> >>>> by
>> >>>>>>>>>> default
>> >>>>>>>>>>>>>>>> AIRFLOW-828  |Open
     |Add maximum size for XComs
>> >>>>>>>>>>>>>>>> AIRFLOW-825  |Open
     |Add Dataflow semantics
>> >>>>>>>>>>>>>>>> AIRFLOW-788  |Open
     |Context unexpectedly added to
>> >>>> hive
>> >>>>>> conf
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I will be locking
down what can get cherry-picked into
>> >>> the
>> >>>>>> 1.9.0
>> >>>>>>>>>>>> branch
>> >>>>>>>>>>>>>>>> shortly, so if you
have something you want in, please
>> >>> set
>> >>>>> the
>> >>>>>>> fix
>> >>>>>>>>>>>>>> version
>> >>>>>>>>>>>>>>>> to 1.9.0.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> We (at WePay) have
deployed 1.9.0 into our dev cluster,
>> >>>> and
>> >>>>> it
>> >>>>>>> has
>> >>>>>>>>>>>> been
>> >>>>>>>>>>>>>>>> running smoothly
for several days.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> ** I could really
use help verifying stability. If you
>> >>> run
>> >>>>>>>>>> Airflow,
>> >>>>>>>>>>>> it's
>> >>>>>>>>>>>>>>>> in your best interest
to deploy the 1.9.0 test branch
>> >>>>>> somewhere,
>> >>>>>>>>>> and
>> >>>>>>>>>>>>>> verify
>> >>>>>>>>>>>>>>>> it's working for
your workload. **
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>>> Chris
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message