spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Blue <rb...@netflix.com.INVALID>
Subject Re: [VOTE] SPARK 2.3.2 (RC6)
Date Fri, 21 Sep 2018 00:09:29 GMT
Changing my vote to +1 with this fixed.

Here's what was going on -- and thanks to Owen O'Malley for debugging:

The problem was that Iceberg contained a fix for a JVM bug for timestamps
before the unix epoch where the timestamp was off by 1s. Owen moved this
code into ORC as well and using the new version of Spark pulled in the
newer version of ORC. That meant that the values were "fixed" twice and
were wrong.

Updating the Iceberg code to rely on the fix in the version of ORC that
Spark includes fixes the problem.

On Thu, Sep 20, 2018 at 2:38 PM Dongjoon Hyun <dongjoon.hyun@gmail.com>
wrote:

> Hi, Ryan.
>
> Could you share the result on 2.3.1 since this is 2.3.2 RC? That would be
> helpful to narrow down the scope.
>
> Bests,
> Dongjoon.
>
> On Thu, Sep 20, 2018 at 11:56 Ryan Blue <rblue@netflix.com.invalid> wrote:
>
>> -0
>>
>> My DataSourceV2 implementation for Iceberg is failing ORC tests when I
>> run with the 2.3.2 RC that pass when I run with 2.3.0. I'm tracking down
>> the cause and will report back, but I'm -0 on the release because there may
>> be a behavior change.
>>
>> On Thu, Sep 20, 2018 at 10:37 AM Denny Lee <denny.g.lee@gmail.com> wrote:
>>
>>> +1
>>>
>>> On Thu, Sep 20, 2018 at 9:55 AM Xiao Li <gatorsmile@gmail.com> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> John Zhuge <john.zhuge@gmail.com> 于2018年9月19日周三 下午1:17写道:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> Built on Ubuntu 16.04 with Maven flags: -Phadoop-2.7 -Pmesos -Pyarn
>>>>> -Phive-thriftserver -Psparkr -Pkinesis-asl -Phadoop-provided
>>>>>
>>>>> java version "1.8.0_181"
>>>>> Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
>>>>>
>>>>>
>>>>> On Wed, Sep 19, 2018 at 2:31 AM Takeshi Yamamuro <
>>>>> linguin.m.s@gmail.com> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> I also checked `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive
>>>>>> -Phive-thriftserve` on the openjdk below/macOSv10.12.6
>>>>>>
>>>>>> $ java -version
>>>>>> java version "1.8.0_181"
>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 19, 2018 at 10:45 AM Dongjoon Hyun <
>>>>>> dongjoon.hyun@gmail.com> wrote:
>>>>>>
>>>>>>> +1.
>>>>>>>
>>>>>>> I tested with `-Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive
>>>>>>> -Phive-thriftserve` on OpenJDK(1.8.0_181)/CentOS 7.5.
>>>>>>>
>>>>>>> I hit the following test case failure once during testing, but
it's
>>>>>>> not persistent.
>>>>>>>
>>>>>>>     KafkaContinuousSourceSuite
>>>>>>>     ...
>>>>>>>     subscribing topic by name from earliest offsets (failOnDataLoss:
>>>>>>> false) *** FAILED ***
>>>>>>>
>>>>>>> Thank you, Saisai.
>>>>>>>
>>>>>>> Bests,
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>> On Mon, Sep 17, 2018 at 6:48 PM Saisai Shao <sai.sai.shao@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 from my own side.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Saisai
>>>>>>>>
>>>>>>>> Wenchen Fan <cloud0fan@gmail.com> 于2018年9月18日周二
上午9:34写道:
>>>>>>>>
>>>>>>>>> +1. All the blocker issues are all resolved in 2.3.2
AFAIK.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 18, 2018 at 9:23 AM Sean Owen <srowen@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 . Licenses and sigs check out as in previous 2.3.x
releases. A
>>>>>>>>>> build from source with most profiles passed for me.
>>>>>>>>>> On Mon, Sep 17, 2018 at 8:17 AM Saisai Shao <
>>>>>>>>>> sai.sai.shao@gmail.com> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Please vote on releasing the following candidate
as Apache
>>>>>>>>>> Spark version 2.3.2.
>>>>>>>>>> >
>>>>>>>>>> > The vote is open until September 21 PST and
passes if a
>>>>>>>>>> majority +1 PMC votes are cast, with a minimum of
3 +1 votes.
>>>>>>>>>> >
>>>>>>>>>> > [ ] +1 Release this package as Apache Spark
2.3.2
>>>>>>>>>> > [ ] -1 Do not release this package because ...
>>>>>>>>>> >
>>>>>>>>>> > To learn more about Apache Spark, please see
>>>>>>>>>> http://spark.apache.org/
>>>>>>>>>> >
>>>>>>>>>> > The tag to be voted on is v2.3.2-rc6 (commit
>>>>>>>>>> 02b510728c31b70e6035ad541bfcdc2b59dcd79a):
>>>>>>>>>> > https://github.com/apache/spark/tree/v2.3.2-rc6
>>>>>>>>>> >
>>>>>>>>>> > The release files, including signatures, digests,
etc. can be
>>>>>>>>>> found at:
>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc6-bin/
>>>>>>>>>> >
>>>>>>>>>> > Signatures used for Spark RCs can be found in
this file:
>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>>>>> >
>>>>>>>>>> > The staging repository for this release can
be found at:
>>>>>>>>>> >
>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1286/
>>>>>>>>>> >
>>>>>>>>>> > The documentation corresponding to this release
can be found at:
>>>>>>>>>> > https://dist.apache.org/repos/dist/dev/spark/v2.3.2-rc6-docs/
>>>>>>>>>> >
>>>>>>>>>> > The list of bug fixes going into 2.3.2 can be
found at the
>>>>>>>>>> following URL:
>>>>>>>>>> > https://issues.apache.org/jira/projects/SPARK/versions/12343289
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > FAQ
>>>>>>>>>> >
>>>>>>>>>> > =========================
>>>>>>>>>> > How can I help test this release?
>>>>>>>>>> > =========================
>>>>>>>>>> >
>>>>>>>>>> > If you are a Spark user, you can help us test
this release by
>>>>>>>>>> taking
>>>>>>>>>> > an existing Spark workload and running on this
release
>>>>>>>>>> candidate, then
>>>>>>>>>> > reporting any regressions.
>>>>>>>>>> >
>>>>>>>>>> > If you're working in PySpark you can set up
a virtual env and
>>>>>>>>>> install
>>>>>>>>>> > the current RC and see if anything important
breaks, in the
>>>>>>>>>> Java/Scala
>>>>>>>>>> > you can add the staging repository to your projects
resolvers
>>>>>>>>>> and test
>>>>>>>>>> > with the RC (make sure to clean up the artifact
cache
>>>>>>>>>> before/after so
>>>>>>>>>> > you don't end up building with a out of date
RC going forward).
>>>>>>>>>> >
>>>>>>>>>> > ===========================================
>>>>>>>>>> > What should happen to JIRA tickets still targeting
2.3.2?
>>>>>>>>>> > ===========================================
>>>>>>>>>> >
>>>>>>>>>> > The current list of open tickets targeted at
2.3.2 can be found
>>>>>>>>>> at:
>>>>>>>>>> > https://issues.apache.org/jira/projects/SPARK
and search for
>>>>>>>>>> "Target Version/s" = 2.3.2
>>>>>>>>>> >
>>>>>>>>>> > Committers should look at those and triage.
Extremely important
>>>>>>>>>> bug
>>>>>>>>>> > fixes, documentation, and API tweaks that impact
compatibility
>>>>>>>>>> should
>>>>>>>>>> > be worked on immediately. Everything else please
retarget to an
>>>>>>>>>> > appropriate release.
>>>>>>>>>> >
>>>>>>>>>> > ==================
>>>>>>>>>> > But my bug isn't fixed?
>>>>>>>>>> > ==================
>>>>>>>>>> >
>>>>>>>>>> > In order to make timely releases, we will typically
not hold the
>>>>>>>>>> > release unless the bug in question is a regression
from the
>>>>>>>>>> previous
>>>>>>>>>> > release. That being said, if there is something
which is a
>>>>>>>>>> regression
>>>>>>>>>> > that has not been correctly targeted please
ping me or a
>>>>>>>>>> committer to
>>>>>>>>>> > help target the issue.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> ---
>>>>>> Takeshi Yamamuro
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> John
>>>>>
>>>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Mime
View raw message