hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakum...@apache.org>
Subject Re: [DISCUSS] ARM/aarch64 support for Hadoop
Date Wed, 30 Oct 2019 10:50:06 GMT
Thanks Zhenyu Zheng for the offer to donate machines for ARM CI.

We will definitely make use of it. Let me check how we can set up Jenkins
job for ARM.

-Thanks
Vinay


On Tue, 22 Oct 2019, 1:04 pm Zhenyu Zheng, <zhengzhenyulixi@gmail.com>
wrote:

> Hi All,
>
> Some updates for ARM CI, our team has succesfully donated ARM resources
> and setup an ARM CI for Apache Spark:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
> it will set to periodic job and then PR trigger when we think it is stable
> enough.
>
> I really hope we can do the same for Hadoop.
>
> BR,
>
> On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vinayakumarb@apache.org>
> wrote:
>
>> Thanks @Anu
>>
>> I understand the concern. I took it in different manner.
>>
>> Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
>> changes as early as possible, its better to do it trunk itself.
>>
>> I was able to come to successfull attempt of upgrading protobuf as per
>> suggestion of stack in Jira
>> https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
>>  .
>>
>> Have created the PR. Please review.  Changes looks huge, because all
>> references of "com.google.protobuf" relocated to
>> "o.a.h.shaded.com.google.protobuf".
>> Otherwise changes are reasonable.
>>
>> This change is with still keeping the current 2.5.0 dependency, for
>> downstream builds. So essentially nothing should be changed for downstreams.
>>
>> -Vinay
>>
>>
>> On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <aengineer@cloudera.com>
>> wrote:
>>
>>> Yes, I think that is what Sunil and I are trying to suggest; the complex
>>> dependencies like Protobuf, if you do it in the trunk you have a better
>>> change of getting it done. Otherwise, at merge point random downstream
>>> applications which you have never heard of will object, and Hadoop
>>> compatibility rules are very clear so you cannot fix it.
>>>
>>> With that said, even doing this in the trunk is complex; It might be
>>> good for you to host a meeting and get some feedback. I have openly said it
>>> is a great idea like "belling the cat", but the effort is in getting the
>>> community to agree and align. Solve that, most of your technical problems
>>> will be easier to solve.
>>>
>>> If you go into a branch, it might be that the community might forget
>>> about your work; and when you come in to merge you will see issues which
>>> you did not think about.
>>>
>>> So, Here is what would be great if you can make this happen; for ARM
>>> work, get a list of dependencies that needed to be upgraded; see if you can
>>> get the community aligned with this goal; since ARM might not be in the
>>> target for many users. You need to convince users that even without ARM,
>>> this is a great idea.
>>>
>>> If you like we can get together during one of the HDFS meetups hosted by
>>> Wei-chiu.
>>>
>>> Thanks
>>> Anu
>>>
>>>
>>>
>>> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vinayakumarb@apache.org>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Thanks for the response.
>>>> As I see, protobuf upgrade is long pending and most awaited one.
>>>>
>>>> @Sunil
>>>> Protobuf upgrade looks to be a non-trivial task.
>>>> Thanks @Duo Zhang for the suggestion of
>>>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>>>> problem
>>>> of dependency on build environment.
>>>> However more problem lies in upgrade protobuf without breaking the
>>>> downstream builds.
>>>> Reason is many downstream projects directly refers server specific jars
>>>> and
>>>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>>>> dependency.
>>>>
>>>> There are some historical discussions and suggestions on
>>>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>>>> upgrade.
>>>> Recommended path for solution is, try to upgrade protobuf using shading
>>>> of
>>>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>>>> dependencies for downstreams.
>>>> I am trying out ideas on this and if it gets completed within time, may
>>>> be
>>>> we can target trunk itself for the protobuf upgrade.
>>>>
>>>> separate branch idea is suggested for the overall ARM support including
>>>> protobuf upgrade and other problems mentioned in the discussion above.
>>>>
>>>> I dont expect separate branch to have a huge changes, apart from bug
>>>> fixes,
>>>> since there are no separate features specific to ARM is being planned.
>>>> So timely rebase of separate branch would reduce the overhead on branch
>>>> review/merge task.
>>>>
>>>> Still, if the solution to protobuf upgrade winds up early, without any
>>>> side
>>>> effects, I am more than happy to land it in trunk itself.
>>>>
>>>> Thanks,
>>>> Vinay
>>>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <sunilg@apache.org>
>>>> wrote:
>>>>
>>>> > Thanks Vinay for starting the thread.
>>>> >
>>>> > I agree to Anu's view point related to protobuf. And with the
>>>> suggestion
>>>> > pointed out by Duo Zhang, if we can make use
>>>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>>>> 3.0.0
>>>> > of protobuf will also be more easier.
>>>> >
>>>> > However i think its better to do this effort in trunk itself.
>>>> > In offline talks, few members were interested to start 3.3.0 release.
>>>> And
>>>> > given that happens soon, I feel its better
>>>> > we do this task in trunk itself as branch diverge is very much
>>>> possible.
>>>> > And to bring to call a merge on such a big branch will be even more
>>>> tough
>>>> > task.
>>>> >
>>>> > my 2 cents.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <palomino219@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin
to
>>>> >> generate
>>>> >> the protobuf code. It will download the protoc binary from the maven
>>>> >> central so we do not need to install protoc on the build machine
any
>>>> more.
>>>> >>
>>>> >> Zhenyu Zheng <zhengzhenyulixi@gmail.com> 于2019年9月4日周三
下午5:27写道:
>>>> >>
>>>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been
>>>> failling
>>>> >> for
>>>> >> > over 2 month related to the Protobuf problem .
>>>> >> > According to the latest successful build log:
>>>> >> >
>>>> >>
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>>>> >> > the
>>>> >> > os version was ubuntu 14.04 and for the jobs that are failling
now
>>>> such
>>>> >> > as:
>>>> https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>>>> >> > the os version is 18.04. I'm not very familiar with the version
>>>> changing
>>>> >> > for the jobs but I did a little search, according to:
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>>>> >> > &
>>>> >> >
>>>> >> >
>>>> >>
>>>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>>>> >> > it both said that the version of libprotc-dev and protobuf-compiler
>>>> >> > available for ubuntu 18.04 is 3.0.0
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ayushtkn@gmail.com>
>>>> wrote:
>>>> >> >
>>>> >> >> Thanx Vinay for the initiative, Makes sense to add support
for
>>>> >> different
>>>> >> >> architectures.
>>>> >> >>
>>>> >> >> +1, for the branch idea.
>>>> >> >> Good Luck!!!
>>>> >> >>
>>>> >> >> -Ayush
>>>> >> >>
>>>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <
>>>> palomino219@gmail.com>
>>>> >> >> wrote:
>>>> >> >> >
>>>> >> >> > For HBase, we purged all the protobuf related things
from the
>>>> public
>>>> >> >> API,
>>>> >> >> > and then upgraded to a shaded and relocated version
of
>>>> protobuf. We
>>>> >> have
>>>> >> >> > created a repo for this:
>>>> >> >> >
>>>> >> >> > https://github.com/apache/hbase-thirdparty
>>>> >> >> >
>>>> >> >> > But since the hadoop dependencies still pull in the
protobuf 2.5
>>>> >> jars,
>>>> >> >> our
>>>> >> >> > coprocessors are still on protobuf 2.5. Recently we
have opened
>>>> a
>>>> >> >> discuss
>>>> >> >> > on how to deal with the upgrading of coprocessor.
Glad to see
>>>> that
>>>> >> the
>>>> >> >> > hadoop community is also willing to solve the problem.
>>>> >> >> >
>>>> >> >> > Anu Engineer <aengineer@cloudera.com.invalid>
于2019年9月3日周二
>>>> 上午1:23写道:
>>>> >> >> >
>>>> >> >> >> +1, for the branch idea. Just FYI, Your biggest
problem is
>>>> proving
>>>> >> that
>>>> >> >> >> Hadoop and the downstream projects work correctly
after you
>>>> upgrade
>>>> >> >> core
>>>> >> >> >> components like Protobuf.
>>>> >> >> >> So while branching and working on a branch is
easy, merging
>>>> back
>>>> >> after
>>>> >> >> you
>>>> >> >> >> upgrade some of these core components is insanely
hard. You
>>>> might
>>>> >> want
>>>> >> >> to
>>>> >> >> >> make sure that community buys into upgrading these
components
>>>> in the
>>>> >> >> trunk.
>>>> >> >> >> That way we will get testing and downstream components
will
>>>> notice
>>>> >> when
>>>> >> >> >> things break.
>>>> >> >> >>
>>>> >> >> >> That said, I have lobbied for the upgrade of Protobuf
for a
>>>> really
>>>> >> long
>>>> >> >> >> time; I have argued that 2.5 is out of support
and we cannot
>>>> stay on
>>>> >> >> that
>>>> >> >> >> branch forever; or we need to take ownership of
the Protobuf
>>>> 2.5
>>>> >> code
>>>> >> >> base.
>>>> >> >> >> It has been rightly pointed to me that while all
the arguments
>>>> I
>>>> >> make
>>>> >> >> is
>>>> >> >> >> correct; it is a very complicated task to upgrade
Protobuf,
>>>> and the
>>>> >> >> worst
>>>> >> >> >> part is we will not even know what breaks until
downstream
>>>> projects
>>>> >> >> pick up
>>>> >> >> >> these changes and work against us.
>>>> >> >> >>
>>>> >> >> >> If we work off the Hadoop version 3 — and assume
that we have
>>>> >> >> "shading" in
>>>> >> >> >> place for all deployments; it might be possible
to get there;
>>>> still
>>>> >> a
>>>> >> >> >> daunting task.
>>>> >> >> >>
>>>> >> >> >> So best of luck with the branch approach — But
please remember,
>>>> >> Merging
>>>> >> >> >> back will be hard, Just my 2 cents.
>>>> >> >> >>
>>>> >> >> >> — Anu
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>>>> >> zhengzhenyulixi@gmail.com
>>>> >> >> >
>>>> >> >> >> wrote:
>>>> >> >> >>
>>>> >> >> >>> Hi,
>>>> >> >> >>>
>>>> >> >> >>> Thanks Vinaya for bring this up and thanks
Sheng for the
>>>> idea. A
>>>> >> >> separate
>>>> >> >> >>> branch with it's own ARM CI seems a really
good idea.
>>>> >> >> >>> By doing this we won't break any of the undergoing
>>>> development in
>>>> >> >> trunk
>>>> >> >> >> and
>>>> >> >> >>> a CI can be a very good way to show what are
the
>>>> >> >> >>> current problems and what have been fixed,
it will also
>>>> provide a
>>>> >> very
>>>> >> >> >> good
>>>> >> >> >>> view for contributors that are intrested to
working on
>>>> >> >> >>> this. We can finally merge back the branch
to trunk until the
>>>> >> >> community
>>>> >> >> >>> thinks it is good enough and stable enough.
We can donate
>>>> >> >> >>> ARM machines to the existing CI system for
the job.
>>>> >> >> >>>
>>>> >> >> >>> I wonder if this approch possible?
>>>> >> >> >>>
>>>> >> >> >>> BR,
>>>> >> >> >>>
>>>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng
Liu <
>>>> >> liusheng2048@gmail.com>
>>>> >> >> >>> wrote:
>>>> >> >> >>>
>>>> >> >> >>>> Hi,
>>>> >> >> >>>>
>>>> >> >> >>>> Thanks Vinay for bring this up, I am a
member of "Openlab"
>>>> >> community
>>>> >> >> >>>> mentioned by Vinay. I am working on building
and
>>>> >> >> >>>> testing Hadoop components on aarch64 server
these days,
>>>> besides
>>>> >> the
>>>> >> >> >>> missing
>>>> >> >> >>>> dependices of ARM platform issues #1 #2
#3
>>>> >> >> >>>> mentioned by Vinay, other similar issue
has also be found,
>>>> such as
>>>> >> >> the
>>>> >> >> >>>> "PhantomJS" dependent package also missing
for aarch64.
>>>> >> >> >>>>
>>>> >> >> >>>> To promote the ARM support for Hadoop,
we have discussed and
>>>> >> hoped to
>>>> >> >> >> add
>>>> >> >> >>>> an ARM specific CI to Hadoop repo. we
are not
>>>> >> >> >>>> sure about if there is any potential effect
or confilict on
>>>> the
>>>> >> trunk
>>>> >> >> >>>> branch, so maybe creating a ARM specific
branch for doing
>>>> these
>>>> >> stuff
>>>> >> >> >>>> is a better choice, what do you think?
>>>> >> >> >>>>
>>>> >> >> >>>> Hope to hear thoughts from you :)
>>>> >> >> >>>>
>>>> >> >> >>>> BR,
>>>> >> >> >>>> Liu sheng
>>>> >> >> >>>>
>>>> >> >> >>>> Vinayakumar B <vinayakumarb@apache.org>
于2019年8月27日周二
>>>> 上午5:34写道:
>>>> >> >> >>>>
>>>> >> >> >>>>> Hi Folks,
>>>> >> >> >>>>>
>>>> >> >> >>>>> ARM is becoming famous lately in its
processing capability
>>>> and
>>>> >> has
>>>> >> >> >> got
>>>> >> >> >>>> the
>>>> >> >> >>>>> potential to run Bigdata workloads.
>>>> >> >> >>>>> Many users have been moving to ARM
machines due to its low
>>>> cost.
>>>> >> >> >>>>>
>>>> >> >> >>>>> In the past there were attempts to
compile Hadoop on ARM
>>>> >> (Rasberry
>>>> >> >> >> PI)
>>>> >> >> >>>> for
>>>> >> >> >>>>> experimental purposes. Today ARM architecture
is taking
>>>> some of
>>>> >> the
>>>> >> >> >>>>> serverside processing as well. So
there will be/is a real
>>>> need of
>>>> >> >> >>> Hadoop
>>>> >> >> >>>> to
>>>> >> >> >>>>> support ARM architecture as well.
>>>> >> >> >>>>>
>>>> >> >> >>>>> There are bunch of users who are trying
out building Hadoop
>>>> on
>>>> >> ARM,
>>>> >> >> >>>> trying
>>>> >> >> >>>>> to add ARM CI to hadoop and facing
issues[1]. Also some
>>>> >> >> >>>>>
>>>> >> >> >>>>> As of today, Hadoop does not compile
on ARM due to below
>>>> issues,
>>>> >> >> >> found
>>>> >> >> >>>> from
>>>> >> >> >>>>> testing done in openlab in [2].
>>>> >> >> >>>>>
>>>> >> >> >>>>> 1. Protobuf :
>>>> >> >> >>>>> -------------------
>>>> >> >> >>>>>     Hadoop project (also some downstream
projects) stuck to
>>>> >> protobuf
>>>> >> >> >>>> 2.5.0
>>>> >> >> >>>>> version, due to backward compatibility
reasons.
>>>> Protobuf-2.5.0 is
>>>> >> >> not
>>>> >> >> >>>> being
>>>> >> >> >>>>> maintained in the community. While
protobuf 3.x is being
>>>> actively
>>>> >> >> >>> adopted
>>>> >> >> >>>>> widely, still protobuf 3.x provides
wire compatibility for
>>>> proto2
>>>> >> >> >>>> messages.
>>>> >> >> >>>>> Due to some compilation issues in
the generated java code,
>>>> which
>>>> >> can
>>>> >> >> >>>> induce
>>>> >> >> >>>>> problems in downstream. Due to this
reason protobuf upgrade
>>>> from
>>>> >> >> >> 2.5.0
>>>> >> >> >>>> was
>>>> >> >> >>>>> not taken up.
>>>> >> >> >>>>> In 3.0.0 onwards, hadoop supports
shading of libraries to
>>>> avoid
>>>> >> >> >>> classpath
>>>> >> >> >>>>> problem in downstream projects.
>>>> >> >> >>>>>    There are patches available to
fix compilation in
>>>> Hadoop. But
>>>> >> >> >> need
>>>> >> >> >>> to
>>>> >> >> >>>>> find a way to upgrade protobuf to
latest version and still
>>>> >> maintain
>>>> >> >> >> the
>>>> >> >> >>>>> downstream's classpath using shading
feature of Hadoop
>>>> build.
>>>> >> >> >>>>>
>>>> >> >> >>>>>     There is a Jira for protobuf upgrade[3]
created even
>>>> before
>>>> >> >> >> shade
>>>> >> >> >>>>> support was added to Hadoop. Now need
to revisit the Jira
>>>> and
>>>> >> >> >> continue
>>>> >> >> >>>>> explore possibilities.
>>>> >> >> >>>>>
>>>> >> >> >>>>> 2. leveldbjni:
>>>> >> >> >>>>> ---------------
>>>> >> >> >>>>>    Current leveldbjni used in YARN
doesnot support ARM
>>>> >> architecture,
>>>> >> >> >>>> need
>>>> >> >> >>>>> to check whether any of the future
versions support ARM and
>>>> can
>>>> >> >> >> hadoop
>>>> >> >> >>>>> upgrade to that version.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> 3. hadoop-yarn-csi's dependency
>>>> 'protoc-gen-grpc-java:1.15.1'
>>>> >> >> >>>>> -------------------------
>>>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does
not provide ARM
>>>> executable by
>>>> >> >> >>> default
>>>> >> >> >>>> in
>>>> >> >> >>>>> the maven repository. Workaround is
to build it locally and
>>>> keep
>>>> >> in
>>>> >> >> >>> local
>>>> >> >> >>>>> maven repository.
>>>> >> >> >>>>> Need to check whether any future versions
of
>>>> >> 'protoc-gen-grpc-java'
>>>> >> >> >> is
>>>> >> >> >>>>> having ARM executable and whether
hadoop-yarn-csi can
>>>> upgrade it?
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Once the compilation issues are solved,
then there might be
>>>> many
>>>> >> >> >> native
>>>> >> >> >>>>> code related issues due to different
architectures.
>>>> >> >> >>>>> So to explore everything, need to
join hands together and
>>>> >> proceed.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> Let us discuss and check, whether
any body else out there
>>>> who
>>>> >> also
>>>> >> >> >> need
>>>> >> >> >>>> the
>>>> >> >> >>>>> support of Hadoop on ARM architectures
and ready to lend
>>>> their
>>>> >> hands
>>>> >> >> >>> and
>>>> >> >> >>>>> time in this work.
>>>> >> >> >>>>>
>>>> >> >> >>>>>
>>>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>>>> >> >> >>>>> [2]
>>>> >> >> >>
>>>> >> >>
>>>> >>
>>>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>>>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>>>> >> >> >>>>>
>>>> >> >> >>>>> -Vinay
>>>> >> >> >>
>>>> >> >>
>>>> >> >
>>>> >>
>>>> >
>>>>
>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message