spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Azuryy Yu <azury...@gmail.com>
Subject Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)
Date Mon, 16 Dec 2013 08:38:44 GMT
Hi Matei,
Thanks for your response. I am using 0.8.1, and yes, It was using
protobuf-2.5. Sorry I made a mistake before this email.

I used -Phadoop2-yarn, so I don't find it using pb-2.5, actually, I should
use -Pnew-yarn.

Thank you Matei.




On Mon, Dec 16, 2013 at 4:22 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> Are you using 0.8.1? It will build with protobuf 2.5 instead of 2.4 as
> long as you make it depend on Hadoop 2.2. But make sure you build it with
> SPARK_HADOOP_VERSION=2.2.0 or whatever.
>
> Spark 0.8.0 doesn’t support Hadoop 2.2 due to this issue.
>
> Matei
>
> On Dec 15, 2013, at 10:25 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
> > Maybe I am not give a clear description. I am runing Spark on yarn.
> instead
> > of Mesos. I just want build Spark with protobuf2.5. I am not care about
> > Mesos.
> >
> > I've changed Spark pom.xml to probobuf2.5 manually.
> >
> >
> >
> >
> > On Mon, Dec 16, 2013 at 2:02 PM, Matei Zaharia <matei.zaharia@gmail.com
> >wrote:
> >
> >> Mesos will almost certainly compile fine with protobuf 2.5. The protobuf
> >> compiler and binary format is forward-compatible across releases, it’s
> just
> >> the Java artifacts that aren’t. You’ll need to ask Mesos to provide a
> >> version with protobuf 2.5, and use that with these versions of Hadoop.
> >>
> >> Matei
> >>
> >> On Dec 15, 2013, at 7:00 PM, Liu, Raymond <raymond.liu@intel.com>
> wrote:
> >>
> >>> That issue is for 0.9's solution.
> >>>
> >>> And if you mean for 0.8.1, when you build against hadoop 2.2 Yarn,
> >> protobuf is already using 2.5.0 instead of 2.4.1. so it will works fine
> >> with hadoop 2.2
> >>> And regarding on 0.8.1 you build against hadoop 2.2 Yarn, while run
> upon
> >> mesos... strange combination, I am not sure, might have problem. If have
> >> problem, you might need to build mesos against 2.5.0, I don't test
> that, if
> >> you got time, mind take a test?
> >>>
> >>> Best Regards,
> >>> Raymond Liu
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Liu, Raymond [mailto:raymond.liu@intel.com]
> >>> Sent: Monday, December 16, 2013 10:48 AM
> >>> To: dev@spark.incubator.apache.org
> >>> Subject: RE: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)
> >>>
> >>> Hi Azuryy
> >>>
> >>> Please Check https://spark-project.atlassian.net/browse/SPARK-995 for
> >> this protobuf version issue
> >>>
> >>> Best Regards,
> >>> Raymond Liu
> >>>
> >>> -----Original Message-----
> >>> From: Azuryy Yu [mailto:azuryyyu@gmail.com]
> >>> Sent: Monday, December 16, 2013 10:30 AM
> >>> To: dev@spark.incubator.apache.org
> >>> Subject: Re: [VOTE] Release Apache Spark 0.8.1-incubating (rc4)
> >>>
> >>> Hi here,
> >>> Do we have plan to upgrade protobuf from 2.4.1 to 2.5.0? PB has some
> >> uncompatable API between these two versions.
> >>> Hadoop-2.x using protobuf-2.5.0
> >>>
> >>>
> >>> but if some guys want to run Spark on mesos, then mesos using
> >>> protobuf-2.4.1 currently. so we may discuss here for a better solution.
> >>>
> >>>
> >>>
> >>> On Mon, Dec 16, 2013 at 7:42 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
> >>>
> >>>> Thanks Patrick.
> >>>> On 16 Dec 2013 02:43, "Patrick Wendell" <pwendell@gmail.com> wrote:
> >>>>
> >>>>> You can checkout the docs mentioned in the vote thread. There is
also
> >>>>> a pre-build binary for hadoop2 that is compiled for YARN 2.2
> >>>>>
> >>>>> - Patrick
> >>>>>
> >>>>> On Sun, Dec 15, 2013 at 4:31 AM, Azuryy Yu <azuryyyu@gmail.com>
> wrote:
> >>>>>> yarn 2.2, not yarn 0.22, I am so sorry.
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Dec 15, 2013 at 8:31 PM, Azuryy Yu <azuryyyu@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>> Spark-0.8.1 supports yarn 0.22 right? where to find the
release
> note?
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sun, Dec 15, 2013 at 3:20 AM, Henry Saputra <
> >>>>> henry.saputra@gmail.com>wrote:
> >>>>>>>
> >>>>>>>> Yeah seems like it. He was ok with our prev release.
> >>>>>>>> Let's wait for his reply
> >>>>>>>>
> >>>>>>>> On Saturday, December 14, 2013, Patrick Wendell wrote:
> >>>>>>>>
> >>>>>>>>> Henry - from that thread it looks like sebb's concern
was
> >>>>>>>>> something different than this.
> >>>>>>>>>
> >>>>>>>>> On Sat, Dec 14, 2013 at 11:08 AM, Henry Saputra
<
> >>>>>>>> henry.saputra@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>> Hi Patrick,
> >>>>>>>>>>
> >>>>>>>>>> Yeap I agree, but technically ASF VOTE release
on source
> >>>>>>>>>> only,
> >>>>> there
> >>>>>>>>>> even debate about it =), so putting it in the
vote staging
> >>>>> artifact
> >>>>>>>>>> could confuse people because in our case we
do package 3rd
> >>>>>>>>>> party libraries in the binary jars.
> >>>>>>>>>>
> >>>>>>>>>> I have sent email to sebb asking clarification
about his
> >>>>>>>>>> concern
> >>>>> in
> >>>>>>>>>> general@ list.
> >>>>>>>>>>
> >>>>>>>>>> - Henry
> >>>>>>>>>>
> >>>>>>>>>> On Sat, Dec 14, 2013 at 10:56 AM, Patrick Wendell
<
> >>>>> pwendell@gmail.com
> >>>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>> Hey Henry,
> >>>>>>>>>>>
> >>>>>>>>>>> One thing a lot of people do during the
vote is test the
> >>>>> binaries and
> >>>>>>>>>>> make sure they work. This is really valuable.
If you'd like
> >>>>>>>>>>> I
> >>>>> could
> >>>>>>>>>>> add a caveat to the vote thread explaining
that we are only
> >>>>> voting on
> >>>>>>>>>>> the source.
> >>>>>>>>>>>
> >>>>>>>>>>> - Patrick
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Dec 14, 2013 at 10:40 AM, Henry
Saputra <
> >>>>>>>>> henry.saputra@gmail.com> wrote:
> >>>>>>>>>>>> Actually we should be fine putting the
binaries there as
> >>>>>>>>>>>> long
> >>>>> as the
> >>>>>>>>>>>> VOTE is for the source.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let's verify with sebb in the general@
list about his
> concern.
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Henry
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Dec 14, 2013 at 10:31 AM, Henry
Saputra <
> >>>>>>>>> henry.saputra@gmail.com> wrote:
> >>>>>>>>>>>>> Hi Patrick, as sebb has mentioned
let's move the binaries
> >>>>>>>>>>>>> from
> >>>>> the
> >>>>>>>>>>>>> voting directory in your people.apache.org
directory.
> >>>>>>>>>>>>> ASF release voting is for source
code and not binaries,
> >>>>>>>>>>>>> and technically we provide binaries
for convenience.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> And add link to the KEYS location
in the dist[1] to let
> >>>>>>>>>>>>> verify
> >>>>>>>>> signatures.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sorry for the late response to the
VOTE thread, guys.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Henry
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1]
> >>>>>>>> https://dist.apache.org/repos/dist/release/incubator/spark/KEYS
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Dec 13, 2013 at 6:37 PM,
Patrick Wendell <
> >>>>>>>> pwendell@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>> The vote is now closed. This
vote passes with 5 PPMC +1's
> >>>>>>>>>>>>>> and
> >>>>> no 0
> >>>>>>>>> or -1
> >>>>>>>>>>>>>> votes.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> +1 (5 Total)
> >>>>>>>>>>>>>> Matei Zaharia*
> >>>>>>>>>>>>>> Nick Pentreath*
> >>>>>>>>>>>>>> Patrick Wendell*
> >>>>>>>>>>>>>> Prashant Sharma*
> >>>>>>>>>>>>>> Tom Graves*
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 0 (0 Total)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -1 (0 Total)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> * = Binding Vote
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As per the incubator release
guide [1] I'll be sending
> >>>>>>>>>>>>>> this
> >>>>> to the
> >>>>>>>>>>>>>> general incubator list for a
final vote from IPMC members.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> http://incubator.apache.org/guides/releasemanagement.html#best-practi
> >>>>> ce-incubator-release-
> >>>>>>>>>>>>>> vote
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Dec 12, 2013 at 8:59
AM, Evan Chan
> >>>>>>>>>>>>>> <ev@ooyala.com>
> >>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'd be personally fine with
a standard workflow of
> >>>>> assemble-deps
> >>>>>>>> +
> >>>>>>>>>>>>>>> packaging just the Spark
files as separate packages, if
> >>>>>>>>>>>>>>> it
> >>>>>>>> speeds up
> >>>>>>>>>>>>>>> everyone's development time.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Dec 11, 2013 at
1:10 PM, Mark Hamstra <
> >>>>>>>>> mark@clearstorydata.com
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I don't know how to
make sense of the numbers, but
> >>>>>>>>>>>>>>>> here's
> >>>>> what
> >>>>>>>>> I've got
> >>>>>>>>>>>>>>>> from a very small sample
size.
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message