hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi Ozawa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14284) Shade Guava everywhere
Date Thu, 04 May 2017 09:28:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996429#comment-15996429
] 

Tsuyoshi Ozawa commented on HADOOP-14284:
-----------------------------------------

[~vinodkv] and [~djp], thanks a lot for your feedback.

> A couple of questions: can we just shade some client jars instead of everywhere? 
> As we keep doing this for other libraries, I'm concerned if our code becomes more brittle
(changing imports everywhere) and if the build times explode.

We're now doing to shade Guava and Curator in hadoop-shaded-thirdparty, and trying to import
it from hadoop-* projects. The build time of Hadoop doesn't get increased so much with this
approach because of just referring to hadoop-shaded-thirdparty project from hadoop-* projects.
However, I found one problem in this approach: shaded artifacts(shaded Guava and Curator)
in hadoop-shaded-thirdparty is NOT in classpath, if I understand correctly. To go with this
approach, we need to unzip source code and compile it like HBase does in hbase-protocol-shaded.
This can make Hadoop build fragile and the build time of Hadoop can increase as Junping Vinod
mentioned. 

https://github.com/apache/hbase/blob/7700a7fac1262934fe538a96b040793c6ff171ce/hbase-protocol-shaded/pom.xml#L321

Gradle seems to have a feature to do this.

http://stackoverflow.com/questions/26244936/how-to-include-only-project-and-relocated-classes-when-using-gradle-shadow-plugi

> Isn't it better to just shade our final artifacts instead of shading individual libraries'
jars? 

Do you mean that we prepare new project "hadoop-server-modules" and shading Guava and Curator
inside them like hadoop-client-modules? It sounds better approach to me. By adding skipShade
option here, we can overcome build time problem. [~andrew.wang] [~busbey] [~ajisakaa] What
do you think?

> Shade Guava everywhere
> ----------------------
>
>                 Key: HADOOP-14284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14284
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 3.0.0-alpha3
>            Reporter: Andrew Wang
>            Assignee: Tsuyoshi Ozawa
>            Priority: Blocker
>         Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch, HADOOP-14284.004.patch,
HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts. Unfortunately, these
projects also consume our private artifacts like {{hadoop-hdfs}}. They also are unlikely on
the new shaded client introduced by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams. This isn't
a requirement for all dependency upgrades, but it's necessary for known-bad dependencies like
Guava.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message