spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-5172) spark-examples-***.jar shades a wrong Hadoop distribution
Date Fri, 09 Jan 2015 10:21:34 GMT
Shixiong Zhu created SPARK-5172:
-----------------------------------

             Summary: spark-examples-***.jar shades a wrong Hadoop distribution
                 Key: SPARK-5172
                 URL: https://issues.apache.org/jira/browse/SPARK-5172
             Project: Spark
          Issue Type: Bug
            Reporter: Shixiong Zhu


Steps to check it:

1. Download  "spark-1.2.0-bin-hadoop2.4.tgz" from http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
2. unzip `spark-assembly-1.2.0-hadoop2.4.0.jar`.
3. There is a file called `org/apache/hadoop/package-info.class` in the jar. It doesn't exist
in hadoop 2.4. 
4. Run "javap -classpath . -private -c -v  org.apache.hadoop.package-info"
{code}
Compiled from "package-info.java"
interface org.apache.hadoop.package-info
  SourceFile: "package-info.java"
  RuntimeVisibleAnnotations: length = 0x24
   00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A
   00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00
   11 73 00 12 
  minor version: 0
  major version: 50
  Constant pool:
const #1 = Asciz	org/apache/hadoop/package-info;
const #2 = class	#1;	//  "org/apache/hadoop/package-info"
const #3 = Asciz	java/lang/Object;
const #4 = class	#3;	//  java/lang/Object
const #5 = Asciz	package-info.java;
const #6 = Asciz	Lorg/apache/hadoop/HadoopVersionAnnotation;;
const #7 = Asciz	version;
const #8 = Asciz	1.2.1;
const #9 = Asciz	revision;
const #10 = Asciz	1503152;
const #11 = Asciz	user;
const #12 = Asciz	mattf;
const #13 = Asciz	date;
const #14 = Asciz	Wed Jul 24 13:39:35 PDT 2013;
const #15 = Asciz	url;
const #16 = Asciz	https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2;
const #17 = Asciz	srcChecksum;
const #18 = Asciz	6923c86528809c4e7e6f493b6b413a9a;
const #19 = Asciz	SourceFile;
const #20 = Asciz	RuntimeVisibleAnnotations;

{
}
{code}
The version is {{1.2.1}}

It comes because a wrong hbase version settings in examples project. Here is a port of the
dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples
dependency:tree"
{noformat}
[INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile
[INFO] |  +- org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile
[INFO] |  +- org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile
[INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.8:compile
[INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.8:compile
[INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
[INFO] |  |  \- com.sun.jersey:jersey-server:jar:1.8:compile
[INFO] |  |     \- asm:asm:jar:3.3.1:test
[INFO] |  +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile
[INFO] |  +- org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile
[INFO] |  +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
[INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  |  \- commons-el:commons-el:jar:1.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile
[INFO] |  |  +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile
[INFO] |  |  +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
[INFO] |  |  +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile
[INFO] |  |  \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile
[INFO] |  +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] |  \- junit:junit:jar:4.10:test
[INFO] |     \- org.hamcrest:hamcrest-core:jar:1.1:test
{noformat}

If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am dependency:tree
-Dhbase.profile=hadoop2`, the dependency tree is right.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message