spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenchen Fan <cloud0...@gmail.com>
Subject Re: [VOTE] SPARK 2.4.0 (RC1)
Date Fri, 21 Sep 2018 05:29:32 GMT
There is an issue in the merge script, when resolving a ticket, the default
fixed version is 3.0.0. I guess someone forgot to type the fixed version
and lead to this mistake.

On Fri, Sep 21, 2018 at 1:15 PM Jungtaek Lim <kabhwan@gmail.com> wrote:

> Ah these issues were resolved before branch-2.4 is cut, like SPARK-24441
>
>
> https://github.com/apache/spark/blob/v2.4.0-rc1/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
>
> SPARK-24441 is included to Spark 2.4.0 RC1 but set to 3.0.0. I heard
> there's a step which version of issues are aligned with new release when
> branch/RC is being cut, but it doesn't look like happening for some issues.
>
> 2018년 9월 21일 (금) 오후 2:10, Holden Karau <holden@pigscanfly.ca>님이
작성:
>
>> So normally during the release process if it's in branch-2.4 but not part
>> of the current RC we set the resolved version to 2.4.1 and then if roll a
>> new RC we switch the 2.4.1 issues to 2.4.0.
>>
>> On Thu, Sep 20, 2018 at 9:55 PM Jungtaek Lim <kabhwan@gmail.com> wrote:
>>
>>> I also noticed there're some fixed issues which are included in
>>> branch-2.4 but its versions are still 3.0.0. Would we want to update
>>> versions to 2.4.0? If we are not planning to run some automations to
>>> correct it, I'm happy to fix them.
>>>
>>> 2018년 9월 20일 (목) 오후 9:22, Weichen Xu <weichen.xu@databricks.com>님이
작성:
>>>
>>>> We need to merge this.
>>>> https://github.com/apache/spark/pull/22492
>>>> Otherwise mleap cannot build against spark 2.4.0
>>>> Thanks!
>>>>
>>>> On Wed, Sep 19, 2018 at 1:16 PM Yinan Li <liyinan926@gmail.com> wrote:
>>>>
>>>>> FYI: SPARK-23200 has been resolved.
>>>>>
>>>>> On Tue, Sep 18, 2018 at 8:49 AM Felix Cheung <
>>>>> felixcheung_m@hotmail.com> wrote:
>>>>>
>>>>>> If we could work on this quickly - it might get on to future RCs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* Stavros Kontopoulos <stavros.kontopoulos@lightbend.com>
>>>>>> *Sent:* Monday, September 17, 2018 2:35 PM
>>>>>> *To:* Yinan Li
>>>>>> *Cc:* Xiao Li; eerlands@redhat.com; vanzin@cloudera.com.invalid;
>>>>>> Sean Owen; Wenchen Fan; dev
>>>>>> *Subject:* Re: [VOTE] SPARK 2.4.0 (RC1)
>>>>>>
>>>>>> Hi Xiao,
>>>>>>
>>>>>> I just tested it, it seems ok. There are some questions about which
>>>>>> properties we should keep when restoring the config. Otherwise it
looks ok
>>>>>> to me.
>>>>>> The reason this should go in 2.4 is that streaming on k8s is
>>>>>> something people want to try day one (or at least it is cool to try)
and
>>>>>> since 2.4 comes with k8s support being refactored a lot,
>>>>>> it would be disappointing not to have it in...IMHO.
>>>>>>
>>>>>> Best,
>>>>>> Stavros
>>>>>>
>>>>>> On Mon, Sep 17, 2018 at 11:13 PM, Yinan Li <liyinan926@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> We can merge the PR and get SPARK-23200 resolved if the whole
point
>>>>>>> is to make streaming on k8s work first. But given that this is
not a
>>>>>>> blocker for 2.4, I think we can take a bit more time here and
get it right.
>>>>>>> With that being said, I would expect it to be resolved soon.
>>>>>>>
>>>>>>> On Mon, Sep 17, 2018 at 11:47 AM Xiao Li <gatorsmile@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, Erik and Stavros,
>>>>>>>>
>>>>>>>> This bug fix SPARK-23200 is not a blocker of the 2.4 release.
It
>>>>>>>> sounds important for the Streaming on K8S. Could the K8S
oriented
>>>>>>>> committers speed up the reviews?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Xiao
>>>>>>>>
>>>>>>>> Erik Erlandson <eerlands@redhat.com> 于2018年9月17日周一
上午11:04写道:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have no binding vote but I second Stavros’ recommendation
for
>>>>>>>>> spark-23200
>>>>>>>>>
>>>>>>>>> Per parallel threads on Py2 support I would also like
to propose
>>>>>>>>> deprecating Py2 starting with this 2.4 release
>>>>>>>>>
>>>>>>>>> On Mon, Sep 17, 2018 at 10:38 AM Marcelo Vanzin
>>>>>>>>> <vanzin@cloudera.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> You can log in to https://repository.apache.org and
see what's
>>>>>>>>>> wrong.
>>>>>>>>>> Just find that staging repo and look at the messages.
In your
>>>>>>>>>> case it
>>>>>>>>>> seems related to your signature.
>>>>>>>>>>
>>>>>>>>>> failureMessageNo public key: Key with id: (xxxx)
was not able to
>>>>>>>>>> be
>>>>>>>>>> located on http://gpg-keyserver.de/. Upload your
public key and
>>>>>>>>>> try
>>>>>>>>>> the operation again.
>>>>>>>>>> On Sun, Sep 16, 2018 at 10:00 PM Wenchen Fan <cloud0fan@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> >
>>>>>>>>>> > I confirmed that
>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1285
>>>>>>>>>> is not accessible. I did it via ./dev/create-release/do-release-docker.sh
>>>>>>>>>> -d /my/work/dir -s publish , not sure what's going
wrong. I didn't see any
>>>>>>>>>> error message during it.
>>>>>>>>>> >
>>>>>>>>>> > Any insights are appreciated! So that I can
fix it in the next
>>>>>>>>>> RC. Thanks!
>>>>>>>>>> >
>>>>>>>>>> > On Mon, Sep 17, 2018 at 11:31 AM Sean Owen <srowen@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> I think one build is enough, but haven't
thought it through.
>>>>>>>>>> The
>>>>>>>>>> >> Hadoop 2.6/2.7 builds are already nearly
redundant. 2.12 is
>>>>>>>>>> probably
>>>>>>>>>> >> best advertised as a 'beta'. So maybe publish
a no-hadoop
>>>>>>>>>> build of it?
>>>>>>>>>> >> Really, whatever's the easy thing to do.
>>>>>>>>>> >> On Sun, Sep 16, 2018 at 10:28 PM Wenchen
Fan <
>>>>>>>>>> cloud0fan@gmail.com> wrote:
>>>>>>>>>> >> >
>>>>>>>>>> >> > Ah I missed the Scala 2.12 build. Do
you mean we should
>>>>>>>>>> publish a Scala 2.12 build this time? Current for
Scala 2.11 we have 3
>>>>>>>>>> builds: with hadoop 2.7, with hadoop 2.6, without
hadoop. Shall we do the
>>>>>>>>>> same thing for Scala 2.12?
>>>>>>>>>> >> >
>>>>>>>>>> >> > On Mon, Sep 17, 2018 at 11:14 AM Sean
Owen <
>>>>>>>>>> srowen@apache.org> wrote:
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> A few preliminary notes:
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> Wenchen for some weird reason when
I hit your key in gpg
>>>>>>>>>> --import, it
>>>>>>>>>> >> >> asks for a passphrase. When I skip
it, it's fine, gpg can
>>>>>>>>>> still verify
>>>>>>>>>> >> >> the signature. No issue there really.
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> The staging repo gives a 404:
>>>>>>>>>> >> >>
>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1285/
>>>>>>>>>> >> >> 404 - Repository "orgapachespark-1285
(staging: open)"
>>>>>>>>>> >> >> [id=orgapachespark-1285] exists
but is not exposed.
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> The (revamped) licenses are OK,
though there are some minor
>>>>>>>>>> glitches
>>>>>>>>>> >> >> in the final release tarballs (my
fault) : there's an extra
>>>>>>>>>> directory,
>>>>>>>>>> >> >> and the source release has both
binary and source licenses.
>>>>>>>>>> I'll fix
>>>>>>>>>> >> >> that. Not strictly necessary to
reject the release over
>>>>>>>>>> those.
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> Last, when I check the staging
repo I'll get my answer,
>>>>>>>>>> but, were you
>>>>>>>>>> >> >> able to build 2.12 artifacts as
well?
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> On Sun, Sep 16, 2018 at 9:48 PM
Wenchen Fan <
>>>>>>>>>> cloud0fan@gmail.com> wrote:
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > Please vote on releasing the
following candidate as
>>>>>>>>>> Apache Spark version 2.4.0.
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > The vote is open until September
20 PST and passes if a
>>>>>>>>>> majority +1 PMC votes are cast, with
>>>>>>>>>> >> >> > a minimum of 3 +1 votes.
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > [ ] +1 Release this package
as Apache Spark 2.4.0
>>>>>>>>>> >> >> > [ ] -1 Do not release this
package because ...
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > To learn more about Apache
Spark, please see
>>>>>>>>>> http://spark.apache.org/
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > The tag to be voted on is
v2.4.0-rc1 (commit
>>>>>>>>>> 1220ab8a0738b5f67dc522df5e3e77ffc83d207a):
>>>>>>>>>> >> >> > https://github.com/apache/spark/tree/v2.4.0-rc1
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > The release files, including
signatures, digests, etc.
>>>>>>>>>> can be found at:
>>>>>>>>>> >> >> >
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc1-bin/
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > Signatures used for Spark
RCs can be found in this file:
>>>>>>>>>> >> >> > https://dist.apache.org/repos/dist/dev/spark/KEYS
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > The staging repository for
this release can be found at:
>>>>>>>>>> >> >> >
>>>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1285/
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > The documentation corresponding
to this release can be
>>>>>>>>>> found at:
>>>>>>>>>> >> >> >
>>>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc1-docs/
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > The list of bug fixes going
into 2.4.0 can be found at
>>>>>>>>>> the following URL:
>>>>>>>>>> >> >> >
>>>>>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > FAQ
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > =========================
>>>>>>>>>> >> >> > How can I help test this release?
>>>>>>>>>> >> >> > =========================
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > If you are a Spark user, you
can help us test this
>>>>>>>>>> release by taking
>>>>>>>>>> >> >> > an existing Spark workload
and running on this release
>>>>>>>>>> candidate, then
>>>>>>>>>> >> >> > reporting any regressions.
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > If you're working in PySpark
you can set up a virtual env
>>>>>>>>>> and install
>>>>>>>>>> >> >> > the current RC and see if
anything important breaks, in
>>>>>>>>>> the Java/Scala
>>>>>>>>>> >> >> > you can add the staging repository
to your projects
>>>>>>>>>> resolvers and test
>>>>>>>>>> >> >> > with the RC (make sure to
clean up the artifact cache
>>>>>>>>>> before/after so
>>>>>>>>>> >> >> > you don't end up building
with a out of date RC going
>>>>>>>>>> forward).
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > ===========================================
>>>>>>>>>> >> >> > What should happen to JIRA
tickets still targeting 2.4.0?
>>>>>>>>>> >> >> > ===========================================
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > The current list of open tickets
targeted at 2.4.0 can be
>>>>>>>>>> found at:
>>>>>>>>>> >> >> > https://issues.apache.org/jira/projects/SPARK
and search
>>>>>>>>>> for "Target Version/s" = 2.4.0
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > Committers should look at
those and triage. Extremely
>>>>>>>>>> important bug
>>>>>>>>>> >> >> > fixes, documentation, and
API tweaks that impact
>>>>>>>>>> compatibility should
>>>>>>>>>> >> >> > be worked on immediately.
Everything else please retarget
>>>>>>>>>> to an
>>>>>>>>>> >> >> > appropriate release.
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > ==================
>>>>>>>>>> >> >> > But my bug isn't fixed?
>>>>>>>>>> >> >> > ==================
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > In order to make timely releases,
we will typically not
>>>>>>>>>> hold the
>>>>>>>>>> >> >> > release unless the bug in
question is a regression from
>>>>>>>>>> the previous
>>>>>>>>>> >> >> > release. That being said,
if there is something which is
>>>>>>>>>> a regression
>>>>>>>>>> >> >> > that has not been correctly
targeted please ping me or a
>>>>>>>>>> committer to
>>>>>>>>>> >> >> > help target the issue.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Marcelo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Stavros Kontopoulos
>>>>>>
>>>>>> *Senior Software Engineer *
>>>>>> *Lightbend, Inc. *
>>>>>>
>>>>>> *p:  +30 6977967274 <%2B1%20650%20678%200020>*
>>>>>> *e: stavros.kontopoulos@lightbend.com* <dave.martin@lightbend.com>
>>>>>>
>>>>>>
>>>>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

Mime
View raw message