spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5172) spark-examples-***.jar shades a wrong Hadoop distribution
Date Fri, 09 Jan 2015 10:34:35 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270853#comment-14270853
] 

Shixiong Zhu commented on SPARK-5172:
-------------------------------------

If someone puts the spark-examples.jar in the classpath, it may use a wrong hadoop client.

> spark-examples-***.jar shades a wrong Hadoop distribution
> ---------------------------------------------------------
>
>                 Key: SPARK-5172
>                 URL: https://issues.apache.org/jira/browse/SPARK-5172
>             Project: Spark
>          Issue Type: Bug
>          Components: Build
>            Reporter: Shixiong Zhu
>            Priority: Minor
>
> Steps to check it:
> 1. Download  "spark-1.2.0-bin-hadoop2.4.tgz" from http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
> 2. unzip `spark-examples-1.2.0-hadoop2.4.0.jar`.
> 3. There is a file called `org/apache/hadoop/package-info.class` in the jar. It doesn't
exist in hadoop 2.4. 
> 4. Run "javap -classpath . -private -c -v  org.apache.hadoop.package-info"
> {code}
> Compiled from "package-info.java"
> interface org.apache.hadoop.package-info
>   SourceFile: "package-info.java"
>   RuntimeVisibleAnnotations: length = 0x24
>    00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A
>    00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00
>    11 73 00 12 
>   minor version: 0
>   major version: 50
>   Constant pool:
> const #1 = Asciz	org/apache/hadoop/package-info;
> const #2 = class	#1;	//  "org/apache/hadoop/package-info"
> const #3 = Asciz	java/lang/Object;
> const #4 = class	#3;	//  java/lang/Object
> const #5 = Asciz	package-info.java;
> const #6 = Asciz	Lorg/apache/hadoop/HadoopVersionAnnotation;;
> const #7 = Asciz	version;
> const #8 = Asciz	1.2.1;
> const #9 = Asciz	revision;
> const #10 = Asciz	1503152;
> const #11 = Asciz	user;
> const #12 = Asciz	mattf;
> const #13 = Asciz	date;
> const #14 = Asciz	Wed Jul 24 13:39:35 PDT 2013;
> const #15 = Asciz	url;
> const #16 = Asciz	https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2;
> const #17 = Asciz	srcChecksum;
> const #18 = Asciz	6923c86528809c4e7e6f493b6b413a9a;
> const #19 = Asciz	SourceFile;
> const #20 = Asciz	RuntimeVisibleAnnotations;
> {
> }
> {code}
> The version is {{1.2.1}}
> It comes because a wrong hbase version settings in examples project. Here is a part of
the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples
dependency:tree"
> {noformat}
> [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile
> [INFO] |  +- org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] |  +- org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.8:compile
> [INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.8:compile
> [INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
> [INFO] |  |  \- com.sun.jersey:jersey-server:jar:1.8:compile
> [INFO] |  |     \- asm:asm:jar:3.3.1:test
> [INFO] |  +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile
> [INFO] |  +- org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] |  +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
> [INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO] |  |  \- commons-el:commons-el:jar:1.0:compile
> [INFO] |  +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile
> [INFO] |  |  +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile
> [INFO] |  |  +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO] |  |  +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile
> [INFO] |  |  \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile
> [INFO] |  +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] |  \- junit:junit:jar:4.10:test
> [INFO] |     \- org.hamcrest:hamcrest-core:jar:1.1:test
> {noformat}
> If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am dependency:tree
-Dhbase.profile=hadoop2`, the dependency tree is right.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message