spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Outstanding Spark 2.1.1 issues
Date Tue, 28 Mar 2017 15:33:14 GMT
We just fixed the build yesterday.  I'll kick off a new RC today.

On Tue, Mar 28, 2017 at 8:04 AM, Asher Krim <akrim@hubspot.com> wrote:

> Hey Michael,
> any update on this? We're itching for a 2.1.1 release (specifically
> SPARK-14804 which is currently blocking us)
>
> Thanks,
> Asher Krim
> Senior Software Engineer
>
> On Wed, Mar 22, 2017 at 7:44 PM, Michael Armbrust <michael@databricks.com>
> wrote:
>
>> An update: I cut the tag for RC1 last night.  Currently fighting with the
>> release process.  Will post RC1 once I get it working.
>>
>> On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <nick.pentreath@gmail.com
>> > wrote:
>>
>>> As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>,
>>> I don't think that needs to be targeted for 2.1.1 so we don't need to worry
>>> about it
>>>
>>>
>>> On Tue, 21 Mar 2017 at 13:49 Holden Karau <holden@pigscanfly.ca> wrote:
>>>
>>>> I agree with Michael, I think we've got some outstanding issues but
>>>> none of them seem like regression from 2.1 so we should be good to start
>>>> the RC process.
>>>>
>>>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <
>>>> michael@databricks.com> wrote:
>>>>
>>>> Please speak up if I'm wrong, but none of these seem like critical
>>>> regressions from 2.1.  As such I'll start the RC process later today.
>>>>
>>>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <holden@pigscanfly.ca>
>>>> wrote:
>>>>
>>>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>>>> regression? Maybe we can get TDs input on it?
>>>>
>>>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zhunanmcgill@gmail.com> wrote:
>>>>
>>>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
>>>> blocker
>>>>
>>>> Best,
>>>>
>>>> Nan
>>>>
>>>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <
>>>> felixcheung_m@hotmail.com> wrote:
>>>>
>>>> I've been scrubbing R and think we are tracking 2 issues
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-19237
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-19925
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* holden.karau@gmail.com <holden.karau@gmail.com> on behalf of
>>>> Holden Karau <holden@pigscanfly.ca>
>>>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>>>> *To:* dev@spark.apache.org
>>>> *Subject:* Outstanding Spark 2.1.1 issues
>>>>
>>>> Hi Spark Developers!
>>>>
>>>> As we start working on the Spark 2.1.1 release I've been looking at our
>>>> outstanding issues still targeted for it. I've tried to break it down by
>>>> component so that people in charge of each component can take a quick look
>>>> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2
&
>>>> the overall list is pretty short (only 9 items - 5 if we only look at
>>>> explicitly tagged) :)
>>>>
>>>> If your working on something for Spark 2.1.1 and it doesn't show up in
>>>> this list please speak up now :) We have a lot of issues (including "in
>>>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>>>> 2.1.1 - if there is something you are working in their which should be
>>>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>>>
>>>> The query string I used for looking at the 2.1.1 open issues is:
>>>>
>>>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>>>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>>>> Unresolved ORDER BY priority DESC
>>>>
>>>> None of the open issues appear to be a regression from 2.1.0, but those
>>>> seem more likely to show up during the RC process (thanks in advance to
>>>> everyone testing their workloads :)) & generally none of them seem to
be
>>>>
>>>> (Note: the cfs are for Target Version/s field)
>>>>
>>>> Critical Issues:
>>>>  SQL:
>>>>   SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> -
Join
>>>> a streaming DataFrame with a batch DataFrame may not work - PR
>>>> https://github.com/apache/spark/pull/17052 (review in progress by
>>>> zsxwing, currently failing Jenkins)*
>>>>
>>>> Major Issues:
>>>>  SQL:
>>>>   SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> -
rand()
>>>> function in case when cause failed - no outstanding PR (consensus on JIRA
>>>> seems to be leaning towards it being a real issue but not necessarily
>>>> everyone agrees just yet - maybe we should slip this?)*
>>>>  Deploy:
>>>>   SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>>>>  - --executor-memory flag doesn't work in local-cluster mode -
>>>> https://github.com/apache/spark/pull/16975 (review in progress by
>>>> vanzin, but PR currently stalled waiting on response) *
>>>>  Core:
>>>>   SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> -
Driver
>>>> fail over will not work, if SPARK_LOCAL* env is set. -
>>>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>>>>  PySpark:
>>>>  SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>>>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>>>> we shouldn't do in a minor release -- but also fixes pip installability
>>>> tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some
>>>> more, but seems like 2.7 support works but some other issues. Maybe slip
to
>>>> 2.2?)
>>>>
>>>> Minor issues:
>>>>  Tests:
>>>>   SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> -
Tests
>>>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>>>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>>>> consider explicitly targeting this for 2.2?
>>>>  PySpark:
>>>>   SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> -
Allow
>>>> to disable hive in pyspark shell - https://github.com/apache/sp
>>>> ark/pull/16906 PR exists but its difficult to add automated tests for
>>>> this (although if SPARK-19955
>>>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
>>>> testing this easier) - no reviewers yet. Possible re-target?*
>>>>  Structured Streaming:
>>>>   SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> -
Flaky
>>>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>>>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>>>> this for 2.2?
>>>>  ML:
>>>>   SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>>>>  - ALSModel.predict on Dataframes : potential optimization by not
>>>> using blas - No PR consider re-targeting unless someone has a PR waiting
in
>>>> the wings?
>>>>
>>>> Explicitly targeted issues are marked with a *, the remaining issues
>>>> are listed as impacting 2.1.1 and don't have a specific target version set.
>>>>
>>>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>>>> blocker in SQL( SPARK-19983
>>>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>>>
>>>> Query string is:
>>>>
>>>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark
>>>> AND resolution = Unresolved AND priority = targetPriority
>>>>
>>>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
>>>> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>>>>
>>>> I'll be going through the 2.1.0 major issues with open PRs that impact
>>>> the PySpark component and seeing if any of them should be targeted for
>>>> 2.1.1, if anyone from the other components wants to take a look through we
>>>> might find some easy wins to be merged.
>>>>
>>>> Cheers,
>>>>
>>>> Holden :)
>>>>
>>>> --
>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>>
>>>> --
>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>
>>
>

Mime
View raw message