spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 赵夏 <zhaoxiah...@gmail.com>
Subject Fwd: Problem about cannot open shared object file
Date Tue, 13 Oct 2015 21:42:18 GMT
Hello Everyone:
  I am new to Spark. Now I meet a problem which I cannot solve by using
Google.
  I have run the Pi example on Hadoop 2.6 successfully.
  After exploiting the spark platform and try to run Pi using spark
"./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
yarn-cluster  --num-executors 2  lib/spark-examples*.jar 10", some problems
occurred.

I listed the error log (found in the userlog directory located at the
slaver node's hadoop directory) below and marked the possible reason to
read color. However, I really do not know how to solve them.



Many thanks
Xia



    136 15/10/13 21:00:19 WARN scheduler.TaskSetManager: Lost task 0.0 in
stage 0.0 (TID 0, slave4): java.io.IOException:
java.lang.reflect.InvocationTargetException
    137     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    138     at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    139     at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    140     at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    141     at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    142     at
org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    143     at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    144     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    145     at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    146     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    147     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    148     at java.lang.Thread.run(Thread.java:745)
    149 Caused by: java.lang.reflect.InvocationTargetException
    150     at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    151     at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    152     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    153     at
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    154     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    155     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    156     at org.apache.spark.broadcast.TorrentBroadcast.org
$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    157     at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    158     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    159     ... 11 more
    160 Caused by: java.lang.IllegalArgumentException:
java.lang.UnsatisfiedLinkError:
/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libs
       nappyjava.so:
/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libsnappyjava.so:
cannot open shared object file: No such file or directory
    161     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    162     ... 20 more
    163 Caused by: java.lang.UnsatisfiedLinkError:
/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/
snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libsnappyjava.so:
/opt/hadoop/tmp/nm-loc
 al-dir/usercache/root/appcache/application_1444764656424_0003/container_1444764656424_0003_02_000003/tmp/
snappy-1.0.4.1-43a64f64-fd65-4f5d-82d0-275eacbd4420-libsnappyjava.so: cannot
open shared object file: No such file or directory
    164     at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    165     at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937)
    166     at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
    167     at java.lang.Runtime.load0(Runtime.java:809)
    168     at java.lang.System.load(System.java:1083)
    169     at
org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:166)
    170     at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145)
    171     at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
    172     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    173     ... 20 more
    174
    175 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Lost task 1.0 in
stage 0.0 (TID 1) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 1]
    176 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Starting task 1.1
in stage 0.0 (TID 3, slave4, PROCESS_LOCAL, 2085 bytes)
    177 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Lost task 2.0 in
stage 0.0 (TID 2) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 2]
    178 15/10/13 21:00:19 INFO scheduler.TaskSetManager: Starting task 2.1
in stage 0.0 (TID 4, slave4, PROCESS_LOCAL, 2085 bytes)
    179 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 1.1 in
stage 0.0 (TID 3) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 3]
    180 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 1.2
in stage 0.0 (TID 5, slave4, PROCESS_LOCAL, 2085 bytes)
    181 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 2.1 in
stage 0.0 (TID 4) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 4]
    182 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 2.2
in stage 0.0 (TID 6, slave4, PROCESS_LOCAL, 2085 bytes)
    183 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 1.2 in
stage 0.0 (TID 5) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 5]
    184 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 1.3
in stage 0.0 (TID 7, slave4, PROCESS_LOCAL, 2085 bytes)
    185 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 2.2 in
stage 0.0 (TID 6) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 6]
    186 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Starting task 2.3
in stage 0.0 (TID 8, slave4, PROCESS_LOCAL, 2085 bytes)
    187 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 1.3 in
stage 0.0 (TID 7) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 7]
    188 15/10/13 21:00:20 ERROR scheduler.TaskSetManager: Task 1 in stage
0.0 failed 4 times; aborting job
    189 15/10/13 21:00:20 INFO cluster.YarnClusterScheduler: Cancelling
stage 0
    190 15/10/13 21:00:20 INFO cluster.YarnClusterScheduler: Stage 0 was
cancelled
    191 15/10/13 21:00:20 INFO scheduler.DAGScheduler: ResultStage 0
(reduce at SparkPi.scala:36) failed in 9.254 s
    192 15/10/13 21:00:20 INFO scheduler.TaskSetManager: Lost task 2.3 in
stage 0.0 (TID 8) on executor slave4: java.io.IOException
(java.lang.reflect.InvocationTargetException) [duplicate 8]
    193 15/10/13 21:00:20 INFO cluster.YarnClusterScheduler: Removed
TaskSet 0.0, whose tasks have all completed, from pool
    194 15/10/13 21:00:20 INFO scheduler.DAGScheduler: Job 0 failed: reduce
at SparkPi.scala:36, took 12.216555 s
    195 15/10/13 21:00:20 ERROR yarn.ApplicationMaster: User class threw
exception: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task
1.3 in stage 0.0 (TID 7, slave4): java.io.IOExceptio        n:
java.lang.reflect.InvocationTargetException
    196     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    197     at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    198     at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    199     at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    200     at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    201     at
org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    202     at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    203     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    204     at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    205     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    206     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    207     at java.lang.Thread.run(Thread.java:745)
    208 Caused by: java.lang.reflect.InvocationTargetException
    209     at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    210     at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    211     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    212     at
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    213     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    214     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    215     at org.apache.spark.broadcast.TorrentBroadcast.org
$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    216     at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    217     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    218     ... 11 more
    219 Caused by: java.lang.IllegalArgumentException:
java.lang.NoClassDefFoundError:
Could not initialize class org.xerial.snappy.Snappy
    220     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    221     ... 20 more
    222 Caused by: java.lang.NoClassDefFoundError: Could not initialize
class org.xerial.snappy.Snappy
    223     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    224     ... 20 more
    225
    226 Driver stacktrace:
    227 org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in
stage 0.0 (TID 7, slave4): java.io.IOException:
java.lang.reflect.InvocationTargetException
    228     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    229     at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    230     at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    231     at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    232     at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    233     at
org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    234     at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    235     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    236     at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    237     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    238     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    239     at java.lang.Thread.run(Thread.java:745)
    240 Caused by: java.lang.reflect.InvocationTargetException
    241     at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    242     at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    243     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    244     at
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    245     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    246     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    247     at org.apache.spark.broadcast.TorrentBroadcast.org
$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    248     at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    249     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    250     ... 11 more
    251 Caused by: java.lang.IllegalArgumentException:
java.lang.NoClassDefFoundError: Could not initialize class
org.xerial.snappy.Snappy
    252     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    253     ... 20 more
    254 Caused by: java.lang.NoClassDefFoundError: Could not initialize
class org.xerial.snappy.Snappy
    255     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    256     ... 20 more
    257
    258 Driver stacktrace:
    259     at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
    260     at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
    261     at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
    262     at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    263     at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    264     at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
    265     at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    266     at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
    267     at scala.Option.foreach(Option.scala:236)
    268     at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
    269     at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
    270     at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
    271     at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
    272     at
org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    273     at
org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
    274     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)
    275     at org.apache.spark.SparkContext.runJob(SparkContext.scala:1942)
    276     at
org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1003)
    277     at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    278     at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    279     at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
    280     at org.apache.spark.rdd.RDD.reduce(RDD.scala:985)
    281     at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:36)
    282     at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
    283     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    284     at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    285     at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    286     at java.lang.reflect.Method.invoke(Method.java:483)
    287     at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:525)
    288 Caused by: java.io.IOException:
java.lang.reflect.InvocationTargetException
    289     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    290     at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    291     at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    292     at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    293     at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    294     at
org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    295     at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    296     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    297     at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    298     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    299     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    300     at java.lang.Thread.run(Thread.java:745)
    301 Caused by: java.lang.reflect.InvocationTargetException
    302     at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    303     at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    304     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    305     at
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    306     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    307     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    308     at org.apache.spark.broadcast.TorrentBroadcast.org
$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    309     at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    310     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    311     ... 11 more
    312 Caused by: java.lang.IllegalArgumentException:
java.lang.NoClassDefFoundError: Could not initialize class
org.xerial.snappy.Snappy
    313     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    314     ... 20 more
    315 Caused by: java.lang.NoClassDefFoundError: Could not initialize
class org.xerial.snappy.Snappy
    316     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    317     ... 20 more
    318 15/10/13 21:00:20 INFO yarn.ApplicationMaster: Final app status:
FAILED, exitCode: 15, (reason: User class threw exception:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 0.0 failed 4 times, most recent failure: Lost task 1.3         in
stage 0.0 (TID 7, slave4): java.io.IOException:
java.lang.reflect.InvocationTargetException
    319     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    320     at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    321     at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    322     at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    323     at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    324     at
org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    325     at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    326     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    327     at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    328     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    329     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    330     at java.lang.Thread.run(Thread.java:745)
    331 Caused by: java.lang.reflect.InvocationTargetException
    332     at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    333     at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    334     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    335     at
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    336     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    337     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    338     at org.apache.spark.broadcast.TorrentBroadcast.org
$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    339     at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    340     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    341     ... 11 more
    342 Caused by: java.lang.IllegalArgumentException:
java.lang.NoClassDefFoundError: Could not initialize class
org.xerial.snappy.Snappy
    343     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    344     ... 20 more
    345 Caused by: java.lang.NoClassDefFoundError: Could not initialize
class org.xerial.snappy.Snappy
    346     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    347     ... 20 more
    348
    349 Driver stacktrace:)
    350 15/10/13 21:00:20 INFO spark.SparkContext: Invoking stop() from
shutdown hook
    351 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/metrics/json,null}
    352 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
    353 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/api,null}
    354 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/,null}
    355 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/static,null}
    356 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
    357 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump,null}
    358 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/json,null}
    359 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors,null}
    360 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment/json,null}
    361 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment,null}
    362 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
    363 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd,null}
    364 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/json,null}
    365 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage,null}
    366 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool/json,null}
    367 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool,null}
    368 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/json,null}
    369 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}
    370 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/json,null}
    371 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages,null}
    372 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}
    373 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}
    374 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}
    375 15/10/13 21:00:20 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs,null}
    376 15/10/13 21:00:20 INFO ui.SparkUI: Stopped Spark web UI at
http://172.16.1.222:58868
    377 15/10/13 21:00:20 INFO scheduler.DAGScheduler: Stopping DAGScheduler
    378 15/10/13 21:00:20 INFO cluster.YarnClusterSchedulerBackend:
Shutting down all executors
    379 15/10/13 21:00:20 INFO cluster.YarnClusterSchedulerBackend: Asking
each executor to shut down
    380 15/10/13 21:00:20 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
    381 15/10/13 21:00:20 INFO yarn.ApplicationMaster$AMEndpoint: Driver
terminated or disconnected! Shutting down. slave4:53105
    382 15/10/13 21:00:20 INFO storage.MemoryStore: MemoryStore cleared
    383 15/10/13 21:00:20 INFO storage.BlockManager: BlockManager stopped
    384 15/10/13 21:00:21 INFO storage.BlockManagerMaster:
BlockManagerMaster stopped
    385 15/10/13 21:00:21 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
    386 15/10/13 21:00:21 INFO spark.SparkContext: Successfully stopped
SparkContext
    387 15/10/13 21:00:21 INFO yarn.ApplicationMaster: Unregistering
ApplicationMaster with FAILED (diag message: User class threw exception:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 0.0 failed 4 times, most recent failure: Lost         task 1.3 in
stage 0.0 (TID 7, slave4): java.io.IOException:
java.lang.reflect.InvocationTargetException
    388     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    389     at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
    390     at
org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
    391     at
org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
    392     at
org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
    393     at
org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
    394     at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
    395     at org.apache.spark.scheduler.Task.run(Task.scala:88)
    396     at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    397     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    398     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    399     at java.lang.Thread.run(Thread.java:745)
   400 Caused by: java.lang.reflect.InvocationTargetException
    401     at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    402     at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    403     at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    404     at
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    405     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:67)
    406     at
org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:60)
    407     at org.apache.spark.broadcast.TorrentBroadcast.org
$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73)
    408     at
org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:167)
    409     at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1175)
    410     ... 11 more
    411 Caused by: java.lang.IllegalArgumentException:
java.lang.NoClassDefFoundError: Could not initialize class
org.xerial.snappy.Snappy
    412     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:151)
    413     ... 20 more
    414 Caused by: java.lang.NoClassDefFoundError: Could not initialize
class org.xerial.snappy.Snappy
    415     at
org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:149)
    416     ... 20 more
    417
    418 Driver stacktrace:)
    419 15/10/13 21:00:21 INFO
remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote
daemon.
    420 15/10/13 21:00:21 INFO
remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down;
proceeding with flushing remote transports.
    421 15/10/13 21:00:21 INFO impl.AMRMClientImpl: Waiting for application
to be successfully unregistered.
    422 15/10/13 21:00:21 INFO yarn.ApplicationMaster: Deleting staging
directory .sparkStaging/application_1444764656424_0003
    423 15/10/13 21:00:21 INFO
remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
    424 15/10/13 21:00:21 INFO util.ShutdownHookManager: Shutdown hook
called
    425 15/10/13 21:00:21 INFO util.ShutdownHookManager: Deleting directory
/opt/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1444764656424_0003/spark-83f57ea5-3469-42a7-94c9-e6fe6ce4d9e4

Mime
View raw message