airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <ash_airflowl...@firemirror.com>
Subject Re: 1.9.0 PR mini-rundown
Date Fri, 08 Sep 2017 09:31:34 GMT
There are a few PRs I would also like to see in this release, I've gone through the open Github
pull requests and picked some out. I've only listed ones that I would accept if I was reviewing
them.

One of these two (one fixes the specific Celery setting I needed, the other allows much more
general customization of Celery config):

- https://github.com/apache/incubator-airflow/pull/2572
- https://github.com/apache/incubator-airflow/pull/2542 or https://github.com/apache/incubator-airflow/pull/2558
(both address the same Jira in slightly different ways.)


https://github.com/apache/incubator-airflow/pull/2578/files looks like it might be an important
fix for a change on master, though is lacking tests covering the bug.

https://github.com/apache/incubator-airflow/pull/2540 `Variable.setdefault` raises an exception
when the variable already exists(!), at least when using JSON.


### Two PRs that look good but aren't quite ready yet

https://github.com/apache/incubator-airflow/pull/2327/files The `--ignore_all_dependencies`
flag wasn't working. (This seems to be failing due to a missing license, but otherwise looks
good and is an important fix. I have asked the author to add the missing license to the new
test file.)

https://github.com/apache/incubator-airflow/pull/2531 Server-side paging of DAGS - helps with
large airflow instances. (Though this is currently marked WIP, so I have asked the author
for clarification on if this is ready or not)

https://github.com/apache/incubator-airflow/pull/2562 - adding the 1.8.2 changelog to master
branch.



### And then some nice to haves/simple changes:

https://github.com/apache/incubator-airflow/pull/2532 Migrate the S3Hook to use boto3.

https://github.com/apache/incubator-airflow/pull/2509/files a tiny refactor to move a calculation
only to where it is needed.

https://github.com/apache/incubator-airflow/pull/2479 fix syntax errors in the docs

https://github.com/apache/incubator-airflow/pull/2405 exposes a new option to `airflow worker`
for `--celery_hostname` to support running multiple workers on the same node.

https://github.com/apache/incubator-airflow/pull/2507/files - properly clears hidden fields
in the Connection UI when changing type.

https://github.com/apache/incubator-airflow/pull/2237/files - Fix issue with Gunicorn 19.4
and upgrade to it. Probably a good idea to not be stuck on older versions of modules? (has
conflicts, but its two one-line changes so looks like it would be easy to resolve at merge
time.)

https://github.com/apache/incubator-airflow/pull/2189/files - replace an instance of hardcoded
`scheduled_` prefix with ID_PREFIX variable. No semantic meaning but seems to be the last
instance of this string that isn't using the variable.

I hope this was helpful.

-ash


> On 7 Sep 2017, at 20:26, Chris Riccomini <criccomini@apache.org> wrote:
> 
> Double negative for the win. Rephrase: Highly unlikely K8s will make 1.9.0.
> 
> On Thu, Sep 7, 2017 at 12:26 PM, Chris Riccomini <criccomini@apache.org>
> wrote:
> 
>> Highly unlikely K8s will not make 1.9.0.
>> 
>> On Thu, Sep 7, 2017 at 12:19 PM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>> 
>>> Depends on how fast it stabilizes...
>>> 
>>> Sent from my iPhone
>>> 
>>>> On 7 Sep 2017, at 21:11, Sergei Iakhnin <llevar@gmail.com> wrote:
>>>> 
>>>> The Kubernetes features would be exciting to have. Do you think these
>>> will
>>>> make it into this release?
>>>> 
>>>>> On Thu, Sep 7, 2017 at 8:22 PM Bolke de Bruin <bdbruin@gmail.com>
>>> wrote:
>>>>> 
>>>>> Would be appreciated!
>>>>> 
>>>>> Op 7 sep. 2017 8:06 p.m. schreef "Chris Riccomini" <
>>> criccomini@apache.org
>>>>>> :
>>>>> 
>>>>>> @Bolke, I can wait to cut the test branch until that PR is merged,
if
>>> you
>>>>>> like.
>>>>>> 
>>>>>> On Thu, Sep 7, 2017 at 10:19 AM, Bolke de Bruin <bdbruin@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> One of my team members is preparing a comprehensive PR that reworks
>>> all
>>>>>>> logging to use the standard framework that was recently introduced.
>>> It
>>>>> is
>>>>>>> expected to be become public tomorrow. So please don't cut too
early
>>>>> ;-).
>>>>>>> In addition it would be nice to have a PR clean up? Some PRs
have
>>> been
>>>>>>> waiting for some time.
>>>>>>> 
>>>>>>> BTW: we will maintain 1.8.X for bug fixes until  1.10/2.0 comes
>>> along.
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Bolke
>>>>>>> 
>>>>>>> Sent from my iPhone
>>>>>>> 
>>>>>>>> On 7 Sep 2017, at 18:55, Chris Riccomini <criccomini@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hey all,
>>>>>>>> 
>>>>>>>> The workflow is documented here:
>>>>>>>> 
>>>>>>>> 
>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow
>>>>>>>> 
>>>>>>>> I'll probably cut the 1.9.0 test branch today or tomorrow,
depending
>>>>> on
>>>>>>>> schedule. I'll let folks know when it's cut via dev@.
>>>>>>>> 
>>>>>>>>> Once the branch is cut does that mean no more PRs will
be included
>>>>> in
>>>>>>> the
>>>>>>>> release, or can some changes still be cherry-picked/merged
in?
>>>>>>>> 
>>>>>>>> We can still cherry-pick PRs in. Very quickly, though, we
begin
>>>>>> rejecting
>>>>>>>> any PRs that aren't bug fixes. It's a big exercise to stabilize
>>> these
>>>>>>>> releases, so I will start rejecting merging non-bug fix PRs
pretty
>>>>>>> quickly.
>>>>>>>> Probably within a couple of weeks.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On Thu, Sep 7, 2017 at 9:31 AM, Ash Berlin-Taylor <
>>>>>>>> ash_airflowlist@firemirror.com> wrote:
>>>>>>>> 
>>>>>>>>> What is the workflow? Once the branch is cut does that
mean no more
>>>>>> PRs
>>>>>>>>> will be included in the release, or can some changes
still be
>>>>>>>>> cherry-picked/merged in?
>>>>>>>>> 
>>>>>>>>> (Is this documented somewhere I can read?)
>>>>>>>>> 
>>>>>>>>> Oh, and thing someone in gitter chat pointed out: the
CHANGELOG on
>>>>>>> master
>>>>>>>>> doesn't include entries for 1.8.2.
>>>>>>>>> 
>>>>>>>>> -ash
>>>>>>>>>> On 7 Sep 2017, at 17:28, Andrew Chen <andrewchen@databricks.com>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Any estimate when branch 1.9 will be cut?
>>>>>>>>>> 
>>>>>>>>>> On Thu, Sep 7, 2017 at 9:25 AM, Maxime Beauchemin
<
>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Go for it! Still trying to wrap up 1.8.2 ...
>>>>>>>>>>> 
>>>>>>>>>>> Max
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Sep 7, 2017 at 8:31 AM, Bolke de Bruin
<
>>> bdbruin@gmail.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Awesome!
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 7 Sep 2017, at 17:04, Chris Riccomini
<
>>> criccomini@apache.org
>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I want to get the ball rolling on 1.9.0,
now that 1.8.2 is out
>>>>> the
>>>>>>>>>>> door.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> @Max, I recall you mentioning driving
this, but if you're burnt
>>>>>> out
>>>>>>>>>>> after
>>>>>>>>>>>>> the 1.8.2 work, Joy and I can drive 1.9.0.
Just let me know.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Chris
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> --
>>>> 
>>>> Sergei
>>> 
>> 
>> 


Mime
View raw message