hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845
Date Mon, 12 Aug 2013 22:19:25 GMT
I've just committed HADOOP-9845 to trunk (only trunk at the moment).

To build trunk now you need protoc 2.5.0 (the build will fail with a
warning if you don't have it).

We'd propagate this to the 2 branches once the precommit build is back to
normal and see things are OK.

Thanks.


On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk use
> protoc 2.5.0.
>
> thx
>
>
> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> I can take care of re-installing 2.4 and installing 2.5 in a different
>> location. This would fix 2.0 branch builds as well.
>> Thoughts?
>>
>> -Giri
>>
>>
>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur <tucu@cloudera.com
>> >wrote:
>>
>> > Giri,
>> >
>> > first of all, thanks for installing protoc 2.5.0.
>> >
>> > I didn't know we were installing them as the only version and not
>> driven by
>> > env/path settings.
>> >
>> > Now we have a bit of a problem, precommit builds are broken because of
>> > mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1).
>> >
>> > We have to options:
>> >
>> > 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out any
>> > follow up issues.
>> > 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and
>> 2.5.0
>> > coexisting
>> >
>> > My take would be to commit HADOOP-9845 in trunk, iron out any issues an
>> > then merge it to the other branches.
>> >
>> > We need to sort this out quickly as precommits are not working.
>> >
>> > I'll wait till 3PM today  for objections to option #1, if none I'll
>> commit
>> > it to trunk.
>> >
>> > Thanks.
>> >
>> > Alejandro
>> >
>> >
>> >
>> > On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan <
>> > gkesavan@hortonworks.com> wrote:
>> >
>> > > Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the default
>> > path.
>> > > If we still need 2.4 I may have to install it. Let me know
>> > >
>> > > -Giri
>> > >
>> > >
>> > > On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur <
>> tucu@cloudera.com
>> > > >wrote:
>> > >
>> > > > thanks giri, how do we set 2.4 or 2.5., what is the path to both so
>> we
>> > > can
>> > > > use and env to set it in the jobs?
>> > > >
>> > > > thx
>> > > >
>> > > > Alejandro
>> > > > (phone typing)
>> > > >
>> > > > On Aug 9, 2013, at 23:10, Giridharan Kesavan <
>> gkesavan@hortonworks.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > build slaves hadoop1-hadoop9 now has libprotoc 2.5.0
>> > > > >
>> > > > >
>> > > > >
>> > > > > -Giri
>> > > > >
>> > > > >
>> > > > > On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan <
>> > > > > gkesavan@hortonworks.com> wrote:
>> > > > >
>> > > > >> Alejandro,
>> > > > >>
>> > > > >> I'm upgrading protobuf on slaves hadoop1-hadoop9.
>> > > > >>
>> > > > >> -Giri
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur <
>> > tucu@cloudera.com
>> > > > >wrote:
>> > > > >>
>> > > > >>> pinging again, I need help from somebody with sudo access
to the
>> > > hadoop
>> > > > >>> jenkins boxes to do this or to get sudo access for a
couple of
>> > hours
>> > > to
>> > > > >>> set
>> > > > >>> up myself.
>> > > > >>>
>> > > > >>> Please!!!
>> > > > >>>
>> > > > >>> thx
>> > > > >>>
>> > > > >>>
>> > > > >>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur <
>> > > tucu@cloudera.com
>> > > > >>>> wrote:
>> > > > >>>
>> > > > >>>> To move forward with this we need protoc 2.5.0 in
the apache
>> > hadoop
>> > > > >>>> jenkins boxes.
>> > > > >>>>
>> > > > >>>> Who can help with this? I assume somebody at Y!,
right?
>> > > > >>>>
>> > > > >>>> Thx
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark <
>> eclark@apache.org>
>> > > > >>> wrote:
>> > > > >>>>
>> > > > >>>>> In HBase land we've pretty well discovered that
we'll need to
>> > have
>> > > > the
>> > > > >>>>> same version of protobuf that the HDFS/Yarn/MR
servers are
>> > running.
>> > > > >>>>> That is to say there are issues with ever having
2.4.x and
>> 2.5.x
>> > on
>> > > > >>>>> the same class path.
>> > > > >>>>>
>> > > > >>>>> Upgrading to 2.5.x would be great, as it brings
some new
>> classes
>> > we
>> > > > >>>>> could use.  With that said HBase is getting pretty
close to a
>> > > rather
>> > > > >>>>> large release (0.96.0 aka The Singularity) so
getting this in
>> > > sooner
>> > > > >>>>> rather than later would be great.  If we could
get this into
>> > 2.1.0
>> > > it
>> > > > >>>>> would be great as that would allow us to have
a pretty easy
>> story
>> > > to
>> > > > >>>>> users with regards to protobuf version.
>> > > > >>>>>
>> > > > >>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee <
>> kihwal@yahoo-inc.com
>> > >
>> > > > >>> wrote:
>> > > > >>>>>> Sorry to hijack the thread but, I also wanted
to mention
>> Avro.
>> > See
>> > > > >>>>> HADOOP-9672.
>> > > > >>>>>> The version we are using has memory leak
and inefficiency
>> > issues.
>> > > > >>> We've
>> > > > >>>>> seen users running into it.
>> > > > >>>>>>
>> > > > >>>>>> Kihwal
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> ________________________________
>> > > > >>>>>> From: Tsuyoshi OZAWA <ozawa.tsuyoshi@gmail.com>
>> > > > >>>>>> To: "common-dev@hadoop.apache.org" <
>> > common-dev@hadoop.apache.org>
>> > > > >>>>>> Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org
>> >;
>> > "
>> > > > >>>>> yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>;
"
>> > > > >>>>> mapreduce-dev@hadoop.apache.org" <
>> > mapreduce-dev@hadoop.apache.org>
>> > > > >>>>>> Sent: Thursday, August 8, 2013 1:59 AM
>> > > > >>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for
the 2.1.0 release,
>> > > > >>>>> HADOOP-9845
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> Hi,
>> > > > >>>>>>
>> > > > >>>>>> About Hadoop, Harsh is dealing with this
problem in
>> HADOOP-9346.
>> > > > >>>>>> For more detail, please see the JIRA ticket:
>> > > > >>>>>> https://issues.apache.org/jira/browse/HADOOP-9346
>> > > > >>>>>>
>> > > > >>>>>> - Tsuyoshi
>> > > > >>>>>>
>> > > > >>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro
Abdelnur <
>> > > > >>> tucu@cloudera.com>
>> > > > >>>>> wrote:
>> > > > >>>>>>> I' like to upgrade to protobuf 2.5.0
for the 2.1.0 release.
>> > > > >>>>>>>
>> > > > >>>>>>> As mentioned in HADOOP-9845, Protobuf
2.5 has significant
>> > > benefits
>> > > > >>> to
>> > > > >>>>>>> justify the upgrade.
>> > > > >>>>>>>
>> > > > >>>>>>> Doing the upgrade now, with the first
beta, will make things
>> > > easier
>> > > > >>> for
>> > > > >>>>>>> downstream projects (like HBase) using
protobuf and adopting
>> > > Hadoop
>> > > > >>> 2.
>> > > > >>>>> If
>> > > > >>>>>>> we do the upgrade later, downstream projects
will have to
>> > > support 2
>> > > > >>>>>>> different versions and they my get in
nasty waters due to
>> > > classpath
>> > > > >>>>> issues.
>> > > > >>>>>>>
>> > > > >>>>>>> I've locally tested the patch in a pseudo
deployment of
>> > > 2.1.0-beta
>> > > > >>>>> branch
>> > > > >>>>>>> and it works fine (something is broken
in trunk in the RPC
>> > layer
>> > > > >>>>> YARN-885).
>> > > > >>>>>>>
>> > > > >>>>>>> Now, to do this it will require a few
things:
>> > > > >>>>>>>
>> > > > >>>>>>> * Make sure protobuf 2.5.0 is available
in the jenkins box
>> > > > >>>>>>> * A follow up email to dev@ aliases indicating
developers
>> > should
>> > > > >>>>> install
>> > > > >>>>>>> locally protobuf 2.5.0
>> > > > >>>>>>>
>> > > > >>>>>>> Thanks.
>> > > > >>>>>>>
>> > > > >>>>>>> --
>> > > > >>>>>>> Alejandro
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> Alejandro
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Alejandro
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Alejandro
>> >
>>
>
>
>
> --
> Alejandro
>



-- 
Alejandro

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message