Hello Everyone:
  I am new to Spark. Now I meet a problem which I cannot solve by using Google. 
  I have run the Pi example on Hadoop 2.6 successfully.
  After exploiting the spark platform and try to run Pi using spark "./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster  --num-executors 2  lib/spark-examples*.jar 10", some problems occurred.

I listed the error log (found in the userlog directory located at the slaver node's hadoop directory) below and marked the possible reason to read color. However, I really do not know how to solve them. 



Many thanks
Xia



    136 15/10/13 21:00:19 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, slave4): java.io.IOException: java.lang.reflect.InvocationTargetException
    137     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    138     at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    139     at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    140     at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    141     at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    142     at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    143     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    144     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    145     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    146     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    147     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    148     at java.lang.Thread.run(Thread.java:745)
    149 Caused by: java.lang.reflect.InvocationTargetException
    150     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    151     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    152     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    153     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    154     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    155     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    156     at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    157     at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    158     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    159     ... 11 more
    160 Caused by: java.lang.IllegalArgumentException: java.lang.UnsatisfiedLinkError: /opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libs        nappyjava.so: /opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libsnappyjava.so: cannot open shared object file: No such file or directory
    161     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    162     ... 20 more
    163 Caused by: java.lang.UnsatisfiedLinkError: /opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libsnappyjava.so: /opt/hadoop/tmp/nm-loc        al-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libsnappyjava.so: cannot open shared object file: No such file or directory
    164     at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    165     at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937)
    166     at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
    167     at java.lang.Runtime.load0(Runtime.java:809)
    168     at java.lang.System.load(System.java:1083)
    169     at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:166)
    170     at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145)
    171     at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
    172     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    173     ... 20 more
    174 
    175 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 1]
    176 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 0.0 (TID 3, slave4, PROCESS_LOCAL, 2085 bytes)
    177 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 2]
    178 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Starting task 2.1 in stage 0.0 (TID 4, slave4, PROCESS_LOCAL, 2085 bytes)
    179 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 1.1 in stage 0.0 (TID 3) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 3]
    180 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 0.0 (TID 5, slave4, PROCESS_LOCAL, 2085 bytes)
    181 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 2.1 in stage 0.0 (TID 4) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 4]
    182 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 2.2 in stage 0.0 (TID 6, slave4, PROCESS_LOCAL, 2085 bytes)
    183 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 5]
    184 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 1.3 in stage 0.0 (TID 7, slave4, PROCESS_LOCAL, 2085 bytes)
    185 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 2.2 in stage 0.0 (TID 6) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 6]
    186 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 2.3 in stage 0.0 (TID 8, slave4, PROCESS_LOCAL, 2085 bytes)
    187 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 1.3 in stage 0.0 (TID 7) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 7]
    188 15/10/13 21:00:20 ERROR scheduler.TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting job
    189 15/10/13 21:00:20 INFO cluster.YarnClusterScheduler: Cancelling stage 0
    190 15/10/13 21:00:20 INFO cluster.YarnClusterScheduler: Stage 0 was cancelled
    191 15/10/13 21:00:20 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) failed in 9.254 s
    192 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 2.3 in stage 0.0 (TID 8) on executor slave4: java.io.IOException (java.lang.reflect.InvocationTargetException) [duplicate 8]
    193 15/10/13 21:00:20 INFO cluster.YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
    194 15/10/13 21:00:20 INFO scheduler.DAGScheduler: Job 0 failed: reduce at SparkPi.scala:36, took 12.216555 s
    195 15/10/13 21:00:20 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 7, slave4): java.io.IOExceptio        n: java.lang.reflect.InvocationTargetException
    196     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    197     at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    198     at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    199     at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    200     at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    201     at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    202     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    203     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    204     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    205     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    206     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    207     at java.lang.Thread.run(Thread.java:745)
    208 Caused by: java.lang.reflect.InvocationTargetException
    209     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    210     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    211     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    212     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    213     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    214     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    215     at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    216     at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    217     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    218     ... 11 more
    219 Caused by: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    220     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    221     ... 20 more
    222 Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    223     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    224     ... 20 more
    225 
    226 Driver stacktrace:
    227 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 7, slave4): java.io.IOException: java.lang.reflect.InvocationTargetException
    228     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    229     at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    230     at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    231     at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    232     at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    233     at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    234     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    235     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    236     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    237     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    238     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    239     at java.lang.Thread.run(Thread.java:745)
    240 Caused by: java.lang.reflect.InvocationTargetException
    241     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    242     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    243     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    244     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    245     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    246     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    247     at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    248     at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    249     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    250     ... 11 more
    251 Caused by: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    252     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    253     ... 20 more
    254 Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    255     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    256     ... 20 more
    257 
    258 Driver stacktrace:
    259     at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
    260     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    261     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
    262     at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    263     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    264     at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
    265     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    266     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    267     at scala.Option.foreach(Option.scala:236)
    268     at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
    269     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
    270     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
    271     at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
    272     at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    273     at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    274     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)
    275     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1942)
    276     at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1003)
    277     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    278     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    279     at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
    280     at org.apache.spark.rdd.RDD.reduce(RDD.scala:985)
    281     at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:36)
    282     at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
    283     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    284     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    285     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    286     at java.lang.reflect.Method.invoke(Method.java:483)
    287     at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:525)
    288 Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
    289     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    290     at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    291     at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    292     at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    293     at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    294     at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    295     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    296     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    297     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    298     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    299     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    300     at java.lang.Thread.run(Thread.java:745)
    301 Caused by: java.lang.reflect.InvocationTargetException
    302     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    303     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    304     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    305     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    306     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    307     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    308     at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    309     at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    310     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    311     ... 11 more
    312 Caused by: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    313     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    314     ... 20 more
    315 Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    316     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    317     ... 20 more
    318 15/10/13 21:00:20 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3         in stage 0.0 (TID 7, slave4): java.io.IOException: java.lang.reflect.InvocationTargetException
    319     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    320     at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    321     at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    322     at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    323     at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    324     at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    325     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    326     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    327     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    328     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    329     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    330     at java.lang.Thread.run(Thread.java:745)
    331 Caused by: java.lang.reflect.InvocationTargetException
    332     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    333     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    334     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    335     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    336     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    337     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    338     at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    339     at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    340     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    341     ... 11 more
    342 Caused by: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    343     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    344     ... 20 more
    345 Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    346     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    347     ... 20 more
    348 
    349 Driver stacktrace:)
    350 15/10/13 21:00:20 INFO spark.SparkContext: Invoking stop() from shutdown hook
    351 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
    352 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
    353 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
    354 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
    355 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
    356 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
    357 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
    358 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
    359 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
    360 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
    361 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
    362 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
    363 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
    364 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
    365 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
    366 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
    367 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
    368 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
    369 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
    370 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
    371 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
    372 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
    373 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
    374 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
    375 15/10/13 21:00:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
    376 15/10/13 21:00:20 INFO ui.SparkUI: Stopped Spark web UI at http://172.16.1.222:58868
    377 15/10/13 21:00:20 INFO scheduler.DAGScheduler: Stopping DAGScheduler
    378 15/10/13 21:00:20 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
    379 15/10/13 21:00:20 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
    380 15/10/13 21:00:20 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
    381 15/10/13 21:00:20 INFO yarn.ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. slave4:53105
    382 15/10/13 21:00:20 INFO storage.MemoryStore: MemoryStore cleared
    383 15/10/13 21:00:20 INFO storage.BlockManager: BlockManager stopped
    384 15/10/13 21:00:21 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
    385 15/10/13 21:00:21 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
    386 15/10/13 21:00:21 INFO spark.SparkContext: Successfully stopped SparkContext
    387 15/10/13 21:00:21 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost         task 1.3 in stage 0.0 (TID 7, slave4): java.io.IOException: java.lang.reflect.InvocationTargetException
    388     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    389     at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    390     at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    391     at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    392     at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    393     at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    394     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    395     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    396     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    397     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    398     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    399     at java.lang.Thread.run(Thread.java:745)
   400 Caused by: java.lang.reflect.InvocationTargetException
    401     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    402     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    403     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    404     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    405     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    406     at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    407     at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    408     at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    409     at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    410     ... 11 more
    411 Caused by: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    412     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    413     ... 20 more
    414 Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
    415     at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    416     ... 20 more
    417 
    418 Driver stacktrace:)
    419 15/10/13 21:00:21 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
    420 15/10/13 21:00:21 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
    421 15/10/13 21:00:21 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
    422 15/10/13 21:00:21 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1444764656424_0003
    423 15/10/13 21:00:21 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
    424 15/10/13 21:00:21 INFO util.ShutdownHookManager: Shutdown hook called
    425 15/10/13 21:00:21 INFO util.ShutdownHookManager: Deleting directory /opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/spark-83f57ea5-3469-42a7-94c9-e6fe6ce4d9e4