spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cgalan <cgalangali...@gmail.com>
Subject Error using spark.driver.userClassPathFirst=true
Date Tue, 01 Sep 2015 11:28:58 GMT
Hi,

When I am submitting a spark job in the mode "yarn-cluster" with the
parameter "spark.driver.userClassPathFirst", my job fails; but if I don't
use this params, my job is concluded with success.. My environment is some
nodes with CDH5.4 and Spark 1.3.0.

Spark submit with fail:
spark-submit --class Main --master yarn-cluster --conf
spark.driver.userClassPathFirst=true --conf
spark.executor.userClassPathFirst=true Main.jar

Spark-submit with success:
spark-submit --class Main --master yarn-cluster Main.jar

Error with Snappy:
java.lang.UnsatisfiedLinkError:
org.xerial.snappy.SnappyNative.maxCompressedLength(I)I
	at org.xerial.snappy.SnappyNative.maxCompressedLength(Native Method)
	at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:316)
	at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
	at
org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:157)
	at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199)
	at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$4.apply(TorrentBroadcast.scala:199)
	at scala.Option.map(Option.scala:145)
	at
org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:199)
	at
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:101)
	at
org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:84)
	at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
	at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
	at
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1051)
	at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:761)
	at org.apache.spark.SparkContext.textFile(SparkContext.scala:589)

My example error code:
    public static void main(String[] args) {
    	
		SparkConf conf = new SparkConf().setAppName("Prueba");
//		conf.setMaster("local"); //Comentar para lanzar en el cluster
		
		JavaSparkContext sc = new JavaSparkContext(conf);
		JavaRDD asd = sc.textFile("text.txt");
		asd.count();
			
        sc.close();
    }

Does anyone have any suggestions? The problem of why I need to use
"spark.driver.userClassPathFirst=true" is because I use commons-cli-1.3.1 in
my project, and the spark classpath have one previous version, so  I create
a shade-jar, but without "spark.driver.userClassPathFirst=true" I have
problems of conflicts with the dependencies between my spark.jar and spark
classpath that one class uses the previous class version of classpath
instead of the last version.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Error-using-spark-driver-userClassPathFirst-true-tp24536.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message