spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hakanilter <hakanil...@gmail.com>
Subject Problem with loading files: Loss was due to java.io.EOFException java.io.EOFException
Date Tue, 20 May 2014 08:11:22 GMT
Hi everyone,

I'm having problems with loading files. Either with java code or
spark-shell, I got same errors when I try to load a text file. 

I added hadoop-client and hadoop-common 2.0.0-cdh4.6.0 as dependencies and
maven-shade-plugin is configured.

I have CDH 4.6.0, spark-0.9.1-bin-cdh4 and JDK7 installed on my nodes.

Here is my code:

public class CountTool {

    public static void main(String[] args) {
        // validate input
        if (args.length != 1) {
            System.err.println("invalid usage!\ncount <input>");
            return;
        }

        // input parameters
        String input = args[0];

        // spark context
        SparkConf conf = new SparkConf()
                .setMaster("spark://server08.mydomain.com:7077")
                .setSparkHome("/opt/spark")
                .setJars(new String[] {"/home/hadoop/dist/mycode.jar"})
                .setAppName(CountTool.class.getSimpleName())
                .set("spark.executor.memory", "1g");

        JavaSparkContext jsc = new JavaSparkContext(conf);

        // fix for cdh4
        jsc.hadoopConfiguration().set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
        jsc.hadoopConfiguration().set("fs.file.impl",
org.apache.hadoop.fs.LocalFileSystem.class.getName());

        // load file and count
        JavaRDD<String> file = jsc.textFile(input);
        long count = file.count();
        System.out.println(String.format("file %s %d lines", input, count));
    }

}

I couldn't load any file either from local disk or hdfs with this code. When
I try to load a file from local it gives "Loss was due to
java.io.EOFException" error:

[hadoop@server08 clustering]$ java -cp /home/hadoop/dist/mycode.jar
com.gg.clustering.spark.CountTool /home/hadoop/scripts/clustering/deals.txt
2014-05-20 10:04:22 INFO  Slf4jLogger:80 - Slf4jLogger started
2014-05-20 10:04:22 INFO  Remoting:74 - Starting remoting
2014-05-20 10:04:23 INFO  Remoting:74 - Remoting started; listening on
addresses :[akka.tcp://spark@server08.mydomain.com:34236]
2014-05-20 10:04:23 INFO  Remoting:74 - Remoting now listens on addresses:
[akka.tcp://spark@server08.mydomain.com:34236]
2014-05-20 10:04:23 INFO  SparkEnv:50 - Registering BlockManagerMaster
2014-05-20 10:04:23 INFO  DiskBlockManager:50 - Created local directory at
/tmp/spark-local-20140520100423-ae57
2014-05-20 10:04:23 INFO  MemoryStore:50 - MemoryStore started with capacity
8.4 GB.
2014-05-20 10:04:23 INFO  ConnectionManager:50 - Bound socket to port 58899
with id = ConnectionManagerId(server08.mydomain.com,58899)
2014-05-20 10:04:23 INFO  BlockManagerMaster:50 - Trying to register
BlockManager
2014-05-20 10:04:23 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server08.mydomain.com:58899 with 8.4 GB RAM
2014-05-20 10:04:23 INFO  BlockManagerMaster:50 - Registered BlockManager
2014-05-20 10:04:23 INFO  HttpServer:50 - Starting HTTP Server
2014-05-20 10:04:23 INFO  Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:04:23 INFO  AbstractConnector:338 - Started
SocketConnector@0.0.0.0:38679
2014-05-20 10:04:23 INFO  HttpBroadcast:50 - Broadcast server started at
http://10.35.74.148:38679
2014-05-20 10:04:23 INFO  SparkEnv:50 - Registering MapOutputTracker
2014-05-20 10:04:23 INFO  HttpFileServer:50 - HTTP File server directory is
/tmp/spark-1e8e7419-9d16-446e-a7e2-5e0e6307f7d0
2014-05-20 10:04:23 INFO  HttpServer:50 - Starting HTTP Server
2014-05-20 10:04:23 INFO  Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:04:23 INFO  AbstractConnector:338 - Started
SocketConnector@0.0.0.0:42538
2014-05-20 10:04:23 INFO  Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage/rdd,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/stage,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/pool,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/environment,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/executors,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/metrics/json,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/static,null}
2014-05-20 10:04:23 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/,null}
2014-05-20 10:04:23 INFO  AbstractConnector:338 - Started
SelectChannelConnector@0.0.0.0:4040
2014-05-20 10:04:23 INFO  SparkUI:50 - Started Spark Web UI at
http://server08.mydomain.com:4040
2014-05-20 10:04:23 WARN  NativeCodeLoader:62 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2014-05-20 10:04:24 INFO  SparkContext:50 - Added JAR
/home/hadoop/dist/mycode.jar at http://10.35.74.148:42538/jars/mycode.jar
with timestamp 1400569464072
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Connecting to master
spark://server08.mydomain.com:7077...
2014-05-20 10:04:24 INFO  MemoryStore:50 - ensureFreeSpace(124827) called
with curMem=0, maxMem=9006219264
2014-05-20 10:04:24 INFO  MemoryStore:50 - Block broadcast_0 stored as
values to memory (estimated size 121.9 KB, free 8.4 GB)
2014-05-20 10:04:24 INFO  SparkDeploySchedulerBackend:50 - Connected to
Spark cluster with app ID app-20140520100424-0005
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/0 on
worker-20140520085014-server02.mydomain.com-54319
(server02.mydomain.com:54319) with 16 cores
2014-05-20 10:04:24 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/0 on hostPort server02.mydomain.com:54319 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/1 on
worker-20140520085014-server06.mydomain.com-33290
(server06.mydomain.com:33290) with 16 cores
2014-05-20 10:04:24 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/1 on hostPort server06.mydomain.com:33290 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/2 on
worker-20140520085014-server03.mydomain.com-50690
(server03.mydomain.com:50690) with 16 cores
2014-05-20 10:04:24 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/2 on hostPort server03.mydomain.com:50690 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/3 on
worker-20140520085014-server04.mydomain.com-41041
(server04.mydomain.com:41041) with 16 cores
2014-05-20 10:04:24 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/3 on hostPort server04.mydomain.com:41041 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/4 on
worker-20140520085014-server07.mydomain.com-44641
(server07.mydomain.com:44641) with 16 cores
2014-05-20 10:04:24 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/4 on hostPort server07.mydomain.com:44641 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100424-0005/5 on
worker-20140520085014-server05.mydomain.com-32908
(server05.mydomain.com:32908) with 16 cores
2014-05-20 10:04:24 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100424-0005/5 on hostPort server05.mydomain.com:32908 with 16
cores, 1024.0 MB RAM
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/0 is now RUNNING
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/1 is now RUNNING
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/2 is now RUNNING
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/3 is now RUNNING
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/4 is now RUNNING
2014-05-20 10:04:24 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100424-0005/5 is now RUNNING
2014-05-20 10:04:24 INFO  FileInputFormat:233 - Total input paths to process
: 1
2014-05-20 10:04:24 INFO  SparkContext:50 - Starting job: count at
CountTool.java:44
2014-05-20 10:04:24 INFO  DAGScheduler:50 - Got job 0 (count at
CountTool.java:44) with 2 output partitions (allowLocal=false)
2014-05-20 10:04:24 INFO  DAGScheduler:50 - Final stage: Stage 0 (count at
CountTool.java:44)
2014-05-20 10:04:24 INFO  DAGScheduler:50 - Parents of final stage: List()
2014-05-20 10:04:24 INFO  DAGScheduler:50 - Missing parents: List()
2014-05-20 10:04:24 INFO  DAGScheduler:50 - Submitting Stage 0 (MappedRDD[1]
at textFile at CountTool.java:43), which has no missing parents
2014-05-20 10:04:24 INFO  DAGScheduler:50 - Submitting 2 missing tasks from
Stage 0 (MappedRDD[1] at textFile at CountTool.java:43)
2014-05-20 10:04:24 INFO  TaskSchedulerImpl:50 - Adding task set 0.0 with 2
tasks
2014-05-20 10:04:25 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server03.mydomain.com:39494/user/Executor#1298593046]
with ID 2
2014-05-20 10:04:25 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 0
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:25 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 4 ms
2014-05-20 10:04:25 INFO  TaskSetManager:50 - Starting task 0.0:1 as TID 1
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:25 INFO  TaskSetManager:50 - Serialized task 0.0:1 as 1629
bytes in 0 ms
2014-05-20 10:04:25 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server02.mydomain.com:59496/user/Executor#1432291773]
with ID 0
2014-05-20 10:04:25 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server06.mydomain.com:39547/user/Executor#78156672]
with ID 1
2014-05-20 10:04:25 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server05.mydomain.com:34135/user/Executor#-777886910]
with ID 5
2014-05-20 10:04:25 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server07.mydomain.com:41161/user/Executor#336519762]
with ID 4
2014-05-20 10:04:25 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server04.mydomain.com:34174/user/Executor#-1382135714]
with ID 3
2014-05-20 10:04:25 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server03.mydomain.com:58285 with 589.2 MB RAM
2014-05-20 10:04:26 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server02.mydomain.com:51425 with 589.2 MB RAM
2014-05-20 10:04:26 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server06.mydomain.com:46383 with 589.2 MB RAM
2014-05-20 10:04:26 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server05.mydomain.com:52167 with 589.2 MB RAM
2014-05-20 10:04:26 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server04.mydomain.com:41308 with 589.2 MB RAM
2014-05-20 10:04:26 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server07.mydomain.com:42993 with 589.2 MB RAM
2014-05-20 10:04:27 WARN  TaskSetManager:62 - Lost TID 1 (task 0.0:1)
2014-05-20 10:04:27 WARN  TaskSetManager:62 - Loss was due to
java.io.EOFException
java.io.EOFException
	at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2744)
	at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1032)
	at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
	at org.apache.hadoop.io.UTF8.readChars(UTF8.java:258)
	at org.apache.hadoop.io.UTF8.readString(UTF8.java:250)
	at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
	at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280)
	at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
	at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:145)
	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
	at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:62)
	at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:193)
	at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
	at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
2014-05-20 10:04:27 WARN  TaskSetManager:62 - Lost TID 0 (task 0.0:0)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 1]
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 2
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 0 ms
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Starting task 0.0:1 as TID 3
on executor 5: server05.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Serialized task 0.0:1 as 1629
bytes in 0 ms
2014-05-20 10:04:27 WARN  TaskSetManager:62 - Lost TID 2 (task 0.0:0)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 2]
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 4
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 0 ms
2014-05-20 10:04:27 WARN  TaskSetManager:62 - Lost TID 4 (task 0.0:0)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 3]
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 5
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1629
bytes in 0 ms
2014-05-20 10:04:27 WARN  TaskSetManager:62 - Lost TID 5 (task 0.0:0)
2014-05-20 10:04:27 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 4]
2014-05-20 10:04:27 ERROR TaskSetManager:66 - Task 0.0:0 failed 4 times;
aborting job
2014-05-20 10:04:27 INFO  DAGScheduler:50 - Failed to run count at
CountTool.java:44
Exception in thread "main" org.apache.spark.SparkException: Job aborted:
Task 0.0:0 failed 4 times (most recent failure: Exception failure:
java.io.EOFException)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
	at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
	at scala.Option.foreach(Option.scala:236)
	at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
	at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


I got same error with hdfs:

[hadoop@server08 clustering]$ java -cp /home/hadoop/dist/mycode.jar
com.gg.clustering.spark.CountTool
hdfs://server01.mydomain.com:/user/hadoop/results/clustering/deals.txt
2014-05-20 10:05:26 INFO  Slf4jLogger:80 - Slf4jLogger started
2014-05-20 10:05:26 INFO  Remoting:74 - Starting remoting
2014-05-20 10:05:26 INFO  Remoting:74 - Remoting started; listening on
addresses :[akka.tcp://spark@server08.mydomain.com:55747]
2014-05-20 10:05:26 INFO  Remoting:74 - Remoting now listens on addresses:
[akka.tcp://spark@server08.mydomain.com:55747]
2014-05-20 10:05:26 INFO  SparkEnv:50 - Registering BlockManagerMaster
2014-05-20 10:05:26 INFO  DiskBlockManager:50 - Created local directory at
/tmp/spark-local-20140520100526-e8cb
2014-05-20 10:05:26 INFO  MemoryStore:50 - MemoryStore started with capacity
8.4 GB.
2014-05-20 10:05:26 INFO  ConnectionManager:50 - Bound socket to port 39356
with id = ConnectionManagerId(server08.mydomain.com,39356)
2014-05-20 10:05:26 INFO  BlockManagerMaster:50 - Trying to register
BlockManager
2014-05-20 10:05:26 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server08.mydomain.com:39356 with 8.4 GB RAM
2014-05-20 10:05:26 INFO  BlockManagerMaster:50 - Registered BlockManager
2014-05-20 10:05:26 INFO  HttpServer:50 - Starting HTTP Server
2014-05-20 10:05:26 INFO  Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:05:26 INFO  AbstractConnector:338 - Started
SocketConnector@0.0.0.0:53241
2014-05-20 10:05:26 INFO  HttpBroadcast:50 - Broadcast server started at
http://10.35.74.148:53241
2014-05-20 10:05:26 INFO  SparkEnv:50 - Registering MapOutputTracker
2014-05-20 10:05:26 INFO  HttpFileServer:50 - HTTP File server directory is
/tmp/spark-ba35f7bd-9741-40e3-a814-37dbb9e2592e
2014-05-20 10:05:26 INFO  HttpServer:50 - Starting HTTP Server
2014-05-20 10:05:26 INFO  Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:05:26 INFO  AbstractConnector:338 - Started
SocketConnector@0.0.0.0:39988
2014-05-20 10:05:26 INFO  Server:266 - jetty-7.x.y-SNAPSHOT
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage/rdd,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/storage,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/stage,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages/pool,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/stages,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/environment,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/executors,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/metrics/json,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/static,null}
2014-05-20 10:05:26 INFO  ContextHandler:744 - started
o.e.j.s.h.ContextHandler{/,null}
2014-05-20 10:05:26 INFO  AbstractConnector:338 - Started
SelectChannelConnector@0.0.0.0:4040
2014-05-20 10:05:26 INFO  SparkUI:50 - Started Spark Web UI at
http://server08.mydomain.com:4040
2014-05-20 10:05:26 WARN  NativeCodeLoader:62 - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2014-05-20 10:05:27 INFO  SparkContext:50 - Added JAR
/home/hadoop/dist/mycode.jar at http://10.35.74.148:39988/jars/mycode.jar
with timestamp 1400569527177
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Connecting to master
spark://server08.mydomain.com:7077...
2014-05-20 10:05:27 INFO  MemoryStore:50 - ensureFreeSpace(124827) called
with curMem=0, maxMem=9006219264
2014-05-20 10:05:27 INFO  MemoryStore:50 - Block broadcast_0 stored as
values to memory (estimated size 121.9 KB, free 8.4 GB)
2014-05-20 10:05:27 INFO  SparkDeploySchedulerBackend:50 - Connected to
Spark cluster with app ID app-20140520100527-0006
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/0 on
worker-20140520085014-server02.mydomain.com-54319
(server02.mydomain.com:54319) with 16 cores
2014-05-20 10:05:27 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/0 on hostPort server02.mydomain.com:54319 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/1 on
worker-20140520085014-server06.mydomain.com-33290
(server06.mydomain.com:33290) with 16 cores
2014-05-20 10:05:27 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/1 on hostPort server06.mydomain.com:33290 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/2 on
worker-20140520085014-server03.mydomain.com-50690
(server03.mydomain.com:50690) with 16 cores
2014-05-20 10:05:27 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/2 on hostPort server03.mydomain.com:50690 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/3 on
worker-20140520085014-server04.mydomain.com-41041
(server04.mydomain.com:41041) with 16 cores
2014-05-20 10:05:27 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/3 on hostPort server04.mydomain.com:41041 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/4 on
worker-20140520085014-server07.mydomain.com-44641
(server07.mydomain.com:44641) with 16 cores
2014-05-20 10:05:27 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/4 on hostPort server07.mydomain.com:44641 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor added:
app-20140520100527-0006/5 on
worker-20140520085014-server05.mydomain.com-32908
(server05.mydomain.com:32908) with 16 cores
2014-05-20 10:05:27 INFO  SparkDeploySchedulerBackend:50 - Granted executor
ID app-20140520100527-0006/5 on hostPort server05.mydomain.com:32908 with 16
cores, 1024.0 MB RAM
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/1 is now RUNNING
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/0 is now RUNNING
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/5 is now RUNNING
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/2 is now RUNNING
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/3 is now RUNNING
2014-05-20 10:05:27 INFO  AppClient$ClientActor:50 - Executor updated:
app-20140520100527-0006/4 is now RUNNING
2014-05-20 10:05:28 INFO  FileInputFormat:233 - Total input paths to process
: 1
2014-05-20 10:05:28 INFO  SparkContext:50 - Starting job: count at
CountTool.java:44
2014-05-20 10:05:28 INFO  DAGScheduler:50 - Got job 0 (count at
CountTool.java:44) with 2 output partitions (allowLocal=false)
2014-05-20 10:05:28 INFO  DAGScheduler:50 - Final stage: Stage 0 (count at
CountTool.java:44)
2014-05-20 10:05:28 INFO  DAGScheduler:50 - Parents of final stage: List()
2014-05-20 10:05:28 INFO  DAGScheduler:50 - Missing parents: List()
2014-05-20 10:05:28 INFO  DAGScheduler:50 - Submitting Stage 0 (MappedRDD[1]
at textFile at CountTool.java:43), which has no missing parents
2014-05-20 10:05:28 INFO  DAGScheduler:50 - Submitting 2 missing tasks from
Stage 0 (MappedRDD[1] at textFile at CountTool.java:43)
2014-05-20 10:05:28 INFO  TaskSchedulerImpl:50 - Adding task set 0.0 with 2
tasks
2014-05-20 10:05:28 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server03.mydomain.com:40916/user/Executor#1474912]
with ID 2
2014-05-20 10:05:28 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 0
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:05:28 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 5 ms
2014-05-20 10:05:28 INFO  TaskSetManager:50 - Starting task 0.0:1 as TID 1
on executor 2: server03.mydomain.com (PROCESS_LOCAL)
2014-05-20 10:05:28 INFO  TaskSetManager:50 - Serialized task 0.0:1 as 1692
bytes in 0 ms
2014-05-20 10:05:28 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server02.mydomain.com:42834/user/Executor#388749186]
with ID 0
2014-05-20 10:05:29 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server05.mydomain.com:49626/user/Executor#1815450843]
with ID 5
2014-05-20 10:05:29 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server03.mydomain.com:49424 with 589.2 MB RAM
2014-05-20 10:05:29 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server07.mydomain.com:35109/user/Executor#-199036165]
with ID 4
2014-05-20 10:05:29 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server06.mydomain.com:35535/user/Executor#-1925679390]
with ID 1
2014-05-20 10:05:29 INFO  SparkDeploySchedulerBackend:50 - Registered
executor:
Actor[akka.tcp://sparkExecutor@server04.mydomain.com:53806/user/Executor#725612946]
with ID 3
2014-05-20 10:05:29 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server02.mydomain.com:35927 with 589.2 MB RAM
2014-05-20 10:05:29 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server05.mydomain.com:57374 with 589.2 MB RAM
2014-05-20 10:05:29 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server07.mydomain.com:50669 with 589.2 MB RAM
2014-05-20 10:05:29 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server06.mydomain.com:33799 with 589.2 MB RAM
2014-05-20 10:05:29 INFO  BlockManagerMasterActor$BlockManagerInfo:50 -
Registering block manager server04.mydomain.com:53457 with 589.2 MB RAM
2014-05-20 10:05:30 WARN  TaskSetManager:62 - Lost TID 1 (task 0.0:1)
2014-05-20 10:05:30 WARN  TaskSetManager:62 - Loss was due to
java.io.EOFException
java.io.EOFException
	at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2744)
	at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1032)
	at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
	at org.apache.hadoop.io.UTF8.readChars(UTF8.java:258)
	at org.apache.hadoop.io.UTF8.readString(UTF8.java:250)
	at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
	at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280)
	at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
	at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:145)
	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:40)
	at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:62)
	at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:193)
	at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
	at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
2014-05-20 10:05:30 WARN  TaskSetManager:62 - Lost TID 0 (task 0.0:0)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 1]
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 2
on executor 2: server03.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 1 ms
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Starting task 0.0:1 as TID 3
on executor 5: server05.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Serialized task 0.0:1 as 1692
bytes in 0 ms
2014-05-20 10:05:30 WARN  TaskSetManager:62 - Lost TID 2 (task 0.0:0)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 2]
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 4
on executor 2: server03.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 0 ms
2014-05-20 10:05:30 WARN  TaskSetManager:62 - Lost TID 4 (task 0.0:0)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 3]
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Starting task 0.0:0 as TID 5
on executor 2: server03.mydomain.com (NODE_LOCAL)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Serialized task 0.0:0 as 1692
bytes in 0 ms
2014-05-20 10:05:30 WARN  TaskSetManager:62 - Lost TID 5 (task 0.0:0)
2014-05-20 10:05:30 INFO  TaskSetManager:50 - Loss was due to
java.io.EOFException [duplicate 4]
2014-05-20 10:05:30 ERROR TaskSetManager:66 - Task 0.0:0 failed 4 times;
aborting job
2014-05-20 10:05:30 INFO  DAGScheduler:50 - Failed to run count at
CountTool.java:44
Exception in thread "main" org.apache.spark.SparkException: Job aborted:
Task 0.0:0 failed 4 times (most recent failure: Exception failure:
java.io.EOFException)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
	at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
	at scala.Option.foreach(Option.scala:236)
	at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
	at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)	 
	
	
spark-shell also gives the same error:

scala> sc.textFile("/home/hadoop/scripts/clustering/deals.txt").count()
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkExecutor@server03.mydomain.com:35256/user/Executor#-1346794213]
with ID 2
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkExecutor@server02.mydomain.com:60855/user/Executor#-1316203186]
with ID 0
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkExecutor@server07.mydomain.com:47643/user/Executor#-556408193]
with ID 4
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkExecutor@server05.mydomain.com:55188/user/Executor#466888766]
with ID 5
14/05/20 10:00:22 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkExecutor@server06.mydomain.com:34672/user/Executor#299947202]
with ID 1
14/05/20 10:00:22 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server03.mydomain.com:57441 with 294.9 MB RAM

14/05/20 10:00:22 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server02.mydomain.com:55847 with 294.9 MB RAM
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server07.mydomain.com:52932 with 294.9 MB RAM
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server05.mydomain.com:45204 with 294.9 MB RAM
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server06.mydomain.com:57226 with 294.9 MB RAM
14/05/20 10:00:23 INFO SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://sparkExecutor@server04.mydomain.com:37144/user/Executor#237933685]
with ID 3
14/05/20 10:00:23 INFO BlockManagerMasterActor$BlockManagerInfo: Registering
block manager server04.mydomain.com:55928 with 294.9 MB RAM
14/05/20 10:00:24 INFO MemoryStore: ensureFreeSpace(78267) called with
curMem=0, maxMem=309225062
14/05/20 10:00:24 INFO MemoryStore: Block broadcast_0 stored as values to
memory (estimated size 76.4 KB, free 294.8 MB)
14/05/20 10:00:24 INFO FileInputFormat: Total input paths to process : 1
14/05/20 10:00:24 INFO SparkContext: Starting job: count at <console>:13
14/05/20 10:00:24 INFO DAGScheduler: Got job 0 (count at <console>:13) with
2 output partitions (allowLocal=false)
14/05/20 10:00:24 INFO DAGScheduler: Final stage: Stage 0 (count at
<console>:13)
14/05/20 10:00:24 INFO DAGScheduler: Parents of final stage: List()
14/05/20 10:00:24 INFO DAGScheduler: Missing parents: List()
14/05/20 10:00:24 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[1] at
textFile at <console>:13), which has no missing parents
14/05/20 10:00:24 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0
(MappedRDD[1] at textFile at <console>:13)
14/05/20 10:00:24 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
14/05/20 10:00:24 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:24 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 5 ms
14/05/20 10:00:24 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on
executor 5: server05.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:24 INFO TaskSetManager: Serialized task 0.0:1 as 1570 bytes
in 1 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 0 (task 0.0:0)
14/05/20 10:00:25 WARN TaskSetManager: Loss was due to
java.io.FileNotFoundException
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist
	at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:468)
	at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:373)
	at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:138)
	at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:334)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:711)
	at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:93)
	at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:54)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:156)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:149)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:64)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
	at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
	at org.apache.spark.scheduler.Task.run(Task.scala:53)
	at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211)
	at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
	at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:0 as TID 2 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 0 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 2 (task 0.0:0)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 1]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:0 as TID 3 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 0 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 1 (task 0.0:1)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 2]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:1 as TID 4 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:1 as 1570 bytes
in 1 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 3 (task 0.0:0)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 3]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:0 as TID 5 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:0 as 1570 bytes
in 1 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 4 (task 0.0:1)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 4]
14/05/20 10:00:25 INFO TaskSetManager: Starting task 0.0:1 as TID 6 on
executor 2: server03.mydomain.com (PROCESS_LOCAL)
14/05/20 10:00:25 INFO TaskSetManager: Serialized task 0.0:1 as 1570 bytes
in 0 ms
14/05/20 10:00:25 WARN TaskSetManager: Lost TID 5 (task 0.0:0)
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 5]
14/05/20 10:00:25 ERROR TaskSetManager: Task 0.0:0 failed 4 times; aborting
job
14/05/20 10:00:25 INFO DAGScheduler: Failed to run count at <console>:13
14/05/20 10:00:25 INFO TaskSetManager: Loss was due to
java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist [duplicate 6]
14/05/20 10:00:25 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
org.apache.spark.SparkException: Job aborted: Task 0.0:0 failed 4 times
(most recent failure: Exception failure: java.io.FileNotFoundException: File
file:/home/hadoop/scripts/clustering/deals.txt does not exist)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
	at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
	at scala.Option.foreach(Option.scala:236)
	at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
	at akka.actor.ActorCell.invoke(ActorCell.scala:456)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
	at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
	

But I can load a hdfs file from scala-shell:

scala>
sc.textFile("hdfs://server01.mydomain.com:/user/hadoop/results/clustering/deals.txt").count()
14/05/20 10:02:57 INFO MemoryStore: ensureFreeSpace(111825) called with
curMem=681339, maxMem=309225062
14/05/20 10:02:57 INFO MemoryStore: Block broadcast_7 stored as values to
memory (estimated size 109.2 KB, free 294.1 MB)
14/05/20 10:02:57 INFO FileInputFormat: Total input paths to process : 1
14/05/20 10:02:57 INFO SparkContext: Starting job: count at <console>:13
14/05/20 10:02:57 INFO DAGScheduler: Got job 6 (count at <console>:13) with
2 output partitions (allowLocal=false)
14/05/20 10:02:57 INFO DAGScheduler: Final stage: Stage 6 (count at
<console>:13)
14/05/20 10:02:57 INFO DAGScheduler: Parents of final stage: List()
14/05/20 10:02:57 INFO DAGScheduler: Missing parents: List()
14/05/20 10:02:57 INFO DAGScheduler: Submitting Stage 6 (MappedRDD[15] at
textFile at <console>:13), which has no missing parents
14/05/20 10:02:57 INFO DAGScheduler: Submitting 2 missing tasks from Stage 6
(MappedRDD[15] at textFile at <console>:13)
14/05/20 10:02:57 INFO TaskSchedulerImpl: Adding task set 6.0 with 2 tasks
14/05/20 10:02:57 INFO TaskSetManager: Starting task 6.0:0 as TID 47 on
executor 2: server03.mydomain.com (NODE_LOCAL)
14/05/20 10:02:57 INFO TaskSetManager: Serialized task 6.0:0 as 1688 bytes
in 0 ms
14/05/20 10:02:57 INFO TaskSetManager: Starting task 6.0:1 as TID 48 on
executor 5: server05.mydomain.com (NODE_LOCAL)
14/05/20 10:02:57 INFO TaskSetManager: Serialized task 6.0:1 as 1688 bytes
in 1 ms
14/05/20 10:02:58 INFO DAGScheduler: Completed ResultTask(6, 0)
14/05/20 10:02:58 INFO TaskSetManager: Finished TID 47 in 454 ms on
server03.mydomain.com (progress: 1/2)
14/05/20 10:02:58 INFO TaskSetManager: Finished TID 48 in 504 ms on
server05.mydomain.com (progress: 2/2)
14/05/20 10:02:58 INFO DAGScheduler: Completed ResultTask(6, 1)
14/05/20 10:02:58 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks
have all completed, from pool
14/05/20 10:02:58 INFO DAGScheduler: Stage 6 (count at <console>:13)
finished in 0.506 s
14/05/20 10:02:58 INFO SparkContext: Job finished: count at <console>:13,
took 0.513880747 s
res8: Long = 101


I also tried to load file into an ArrayList and use parallelize(), it worked
too. But I don't understand why I can't just load a file and count it? Could
someone please help me understand the problem? 

Thanks in advance...





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-loading-files-Loss-was-due-to-java-io-EOFException-java-io-EOFException-tp6090.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message