spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <dbt...@dbtsai.com.INVALID>
Subject Re: [VOTE] Release Apache Spark 2.4.1 (RC2)
Date Thu, 07 Mar 2019 18:52:34 GMT
Saisai,

There is no blocker now. I ran into some difficulties in publishing the
jars into Nexus. The publish task was finished, but Nexus gave me the
following error.


*failureMessage Failed to validate the pgp signature of
'/org/apache/spark/spark-streaming-flume-assembly_2.11/2.4.1/spark-streaming-flume-assembly_2.11-2.4.1-tests.jar',
check the logs.*

I am sure my key is in the key server, and the weird thing is that it fails
on different jars each time I ran the publish script.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1


On Wed, Mar 6, 2019 at 6:04 PM Saisai Shao <sai.sai.shao@gmail.com> wrote:

> Do we have other block/critical issues for Spark 2.4.1 or waiting
> something to be fixed? I roughly searched the JIRA, seems there's no
> block/critical issues marked for 2.4.1.
>
> Thanks
> Saisai
>
> shane knapp <sknapp@berkeley.edu> 于2019年3月7日周四 上午4:57写道:
>
>> i'll be popping in to the sig-big-data meeting on the 20th to talk about
>> stuff like this.
>>
>> On Wed, Mar 6, 2019 at 12:40 PM Stavros Kontopoulos <
>> stavros.kontopoulos@lightbend.com> wrote:
>>
>>> Yes its a touch decision and as we discussed today (
>>> https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA
>>> )
>>> "Kubernetes support window is 9 months, Spark is two years". So we may
>>> end up with old client versions on branches still supported like 2.4.x in
>>> the future.
>>> That gives us no choice but to upgrade, if we want to be on the safe
>>> side. We have tested 3.0.0 with 1.11 internally and it works but I dont
>>> know what it means to run with old
>>> clients.
>>>
>>>
>>> On Wed, Mar 6, 2019 at 7:54 PM Sean Owen <srowen@gmail.com> wrote:
>>>
>>>> If the old client is basically unusable with the versions of K8S
>>>> people mostly use now, and the new client still works with older
>>>> versions, I could see including this in 2.4.1.
>>>>
>>>> Looking at
>>>> https://github.com/fabric8io/kubernetes-client#compatibility-matrix
>>>> it seems like the 4.1.1 client is needed for 1.10 and above. However
>>>> it no longer supports 1.7 and below.
>>>> We have 3.0.x, and versions through 4.0.x of the client support the
>>>> same K8S versions, so no real middle ground here.
>>>>
>>>> 1.7.0 came out June 2017, it seems. 1.10 was March 2018. Minor release
>>>> branches are maintained for 9 months per
>>>> https://kubernetes.io/docs/setup/version-skew-policy/
>>>>
>>>> Spark 2.4.0 came in Nov 2018. I suppose we could say it should have
>>>> used the newer client from the start as at that point (?) 1.7 and
>>>> earlier were already at least 7 months past EOL.
>>>> If we update the client in 2.4.1, versions of K8S as recently
>>>> 'supported' as a year ago won't work anymore. I'm guessing there are
>>>> still 1.7 users out there? That wasn't that long ago but if the
>>>> project and users generally move fast, maybe not.
>>>>
>>>> Normally I'd say, that's what the next minor release of Spark is for;
>>>> update if you want later infra. But there is no Spark 2.5.
>>>> I presume downstream distros could modify the dependency easily (?) if
>>>> needed and maybe already do. It wouldn't necessarily help end users.
>>>>
>>>> Does the 3.0.x client not work at all with 1.10+ or just unsupported.
>>>> If it 'basically works but no guarantees' I'd favor not updating. If
>>>> it doesn't work at all, hm. That's tough. I think I'd favor updating
>>>> the client but think it's a tough call both ways.
>>>>
>>>>
>>>>
>>>> On Wed, Mar 6, 2019 at 11:14 AM Stavros Kontopoulos
>>>> <stavros.kontopoulos@lightbend.com> wrote:
>>>> >
>>>> > Yes Shane Knapp has done the work for that already,  and also tests
>>>> pass, I am working on a PR now, I could submit it for the 2.4 branch .
>>>> > I understand that this is a major dependency update, but the problem
>>>> I see is that the client version is so old that I dont think it makes
>>>> > much sense for current users who are on k8s 1.10, 1.11 etc(
>>>> https://github.com/fabric8io/kubernetes-client#compatibility-matrix,
>>>> 3.0.0 does not even exist in there).
>>>> > I dont know what it means to use that old version with current k8s
>>>> clusters in terms of bugs etc.
>>>>
>>>
>>>
>>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>

Mime
View raw message