spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Röwer <paul.roewer1...@googlemail.com>
Subject Re: spark no output
Date Sun, 02 Aug 2015 16:02:06 GMT
hi, what reason have this exception?

Log Type: stderr

Log Upload Time: Sun Aug 02 13:35:16 +0200 2015

Log Length: 72197

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/yarn/nm/usercache/marcel/filecache/296/spark-assembly-1.3.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/08/02 13:25:38 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1438417242433_0206_000001
15/08/02 13:25:38 INFO spark.SecurityManager: Changing view acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: Changing modify acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, marcel); users with modify permissions: Set(yarn, marcel)
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Waiting for spark context initialization
15/08/02 13:25:38 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
15/08/02 13:25:38 INFO spark.SparkContext: Running Spark version 1.3.0
15/08/02 13:25:38 INFO spark.SecurityManager: Changing view acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: Changing modify acls to: yarn,marcel
15/08/02 13:25:38 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, marcel); users with modify permissions: Set(yarn, marcel)
15/08/02 13:25:39 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/08/02 13:25:39 INFO Remoting: Starting remoting
15/08/02 13:25:39 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@node2-scads05:54443]
15/08/02 13:25:39 INFO util.Utils: Successfully started service 'sparkDriver' on port 54443.
15/08/02 13:25:39 INFO spark.SparkEnv: Registering MapOutputTracker
15/08/02 13:25:39 INFO spark.SparkEnv: Registering BlockManagerMaster
15/08/02 13:25:39 INFO storage.DiskBlockManager: Created local directory at /yarn/nm/usercache/marcel/appcache/application_1438417242433_0206/blockmgr-74d8b61c-430c-4d02-bb56-f3fe85c8df1b
15/08/02 13:25:39 INFO storage.MemoryStore: MemoryStore started with capacity 245.7 MB
15/08/02 13:25:39 INFO spark.HttpFileServer: HTTP File server directory is /yarn/nm/usercache/marcel/appcache/application_1438417242433_0206/httpd-7bcf27d9-42ff-4ac3-9296-39afffd559d1
15/08/02 13:25:39 INFO spark.HttpServer: Starting HTTP Server
15/08/02 13:25:39 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/08/02 13:25:39 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:42691
15/08/02 13:25:39 INFO util.Utils: Successfully started service 'HTTP file server' on port 42691.
15/08/02 13:25:39 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/08/02 13:25:39 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/08/02 13:25:39 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/08/02 13:25:39 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:41412
15/08/02 13:25:39 INFO util.Utils: Successfully started service 'SparkUI' on port 41412.
15/08/02 13:25:39 INFO ui.SparkUI: Started SparkUI at http://node2-scads05:41412
15/08/02 13:25:39 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
15/08/02 13:25:39 INFO netty.NettyBlockTransferService: Server created on 50154
15/08/02 13:25:39 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/08/02 13:25:39 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:50154 with 245.7 MB RAM, BlockManagerId(<driver>, node2-scads05, 50154)
15/08/02 13:25:39 INFO storage.BlockManagerMaster: Registered BlockManager
15/08/02 13:25:39 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@node2-scads05:54443/user/YarnScheduler
15/08/02 13:25:39 INFO cluster.YarnClusterSchedulerBackend: ApplicationMaster registered as Actor[akka://sparkDriver/user/YarnAM#347751435]
15/08/02 13:25:39 INFO client.RMProxy: Connecting to ResourceManager at node1-scads02/139.18.13.42:8030
15/08/02 13:25:39 INFO yarn.YarnRMClient: Registering the ApplicationMaster
15/08/02 13:25:39 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:25:39 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:25:39 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:25:39 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
15/08/02 13:25:39 INFO impl.AMRMClientImpl: Received new token for : node3-scads06:8041
15/08/02 13:25:39 INFO impl.AMRMClientImpl: Received new token for : node2-scads05:8041
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000002 for on host node3-scads06
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000003 for on host node2-scads05
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:25:39 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:25:39 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000003/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000003/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000002/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000002/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 1, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:25:39 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 2, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:25:39 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:35454/user/Executor#-243069105] with ID 2
15/08/02 13:25:41 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:34516 with 3.3 GB RAM, BlockManagerId(2, node2-scads05, 34516)
15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:58198/user/Executor#169360033] with ID 1
15/08/02 13:25:41 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
15/08/02 13:25:41 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
15/08/02 13:25:41 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:58512 with 3.3 GB RAM, BlockManagerId(1, node3-scads06, 58512)
15/08/02 13:25:41 INFO storage.MemoryStore: ensureFreeSpace(115917) called with curMem=0, maxMem=257635123
15/08/02 13:25:41 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 113.2 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(26058) called with curMem=115917, maxMem=257635123
15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 25.4 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:50154 (size: 25.4 KB, free: 245.7 MB)
15/08/02 13:25:42 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
15/08/02 13:25:42 INFO spark.SparkContext: Created broadcast 0 from textFile at SparkBenchmarkLauncher.java:71
15/08/02 13:25:42 INFO mapred.FileInputFormat: Total input paths to process : 2
15/08/02 13:25:42 INFO spark.SparkContext: Starting job: collect at SparkBenchmarkLauncher.java:246
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Registering RDD 4 (mapToPair at SparkBenchmarkLauncher.java:181)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Got job 0 (collect at SparkBenchmarkLauncher.java:246) with 4 output partitions (allowLocal=false)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Final stage: Stage 1(collect at SparkBenchmarkLauncher.java:246)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Parents of final stage: List(Stage 0)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Missing parents: List(Stage 0)
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[4] at mapToPair at SparkBenchmarkLauncher.java:181), which has no missing parents
15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(5640) called with curMem=141975, maxMem=257635123
15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.5 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.MemoryStore: ensureFreeSpace(3310) called with curMem=147615, maxMem=257635123
15/08/02 13:25:42 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.2 KB, free 245.6 MB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:50154 (size: 3.2 KB, free: 245.7 MB)
15/08/02 13:25:42 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
15/08/02 13:25:42 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839
15/08/02 13:25:42 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 0 (MapPartitionsRDD[4] at mapToPair at SparkBenchmarkLauncher.java:181)
15/08/02 13:25:42 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 4 tasks
15/08/02 13:25:42 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:25:42 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:58512 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:34516 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:34516 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:25:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:58512 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:25:44 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:34516 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:25:45 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:58512 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:26:40 ERROR cluster.YarnClusterScheduler: Lost executor 1 on node3-scads06: remote Akka client disassociated
15/08/02 13:26:40 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:58198] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:26:40 INFO scheduler.TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0
15/08/02 13:26:40 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, node3-scads06): ExecutorLostFailure (executor 1 lost)
15/08/02 13:26:40 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 1
15/08/02 13:26:40 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 0)
15/08/02 13:26:40 INFO storage.BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster.
15/08/02 13:26:40 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(1, node3-scads06, 58512)
15/08/02 13:26:40 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
15/08/02 13:26:44 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000003 (state: COMPLETE, exit status: -104)
15/08/02 13:26:44 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:26:44 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000002 (state: COMPLETE, exit status: -104)
15/08/02 13:26:44 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.0 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:26:44 ERROR cluster.YarnClusterScheduler: Lost executor 2 on node2-scads05: remote Akka client disassociated
15/08/02 13:26:44 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:35454] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:26:44 INFO scheduler.TaskSetManager: Re-queueing tasks for 2 from TaskSet 0.0
15/08/02 13:26:44 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, node2-scads05): ExecutorLostFailure (executor 2 lost)
15/08/02 13:26:44 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 2
15/08/02 13:26:44 INFO scheduler.DAGScheduler: Executor lost: 2 (epoch 1)
15/08/02 13:26:44 INFO storage.BlockManagerMasterActor: Trying to remove executor 2 from BlockManagerMaster.
15/08/02 13:26:44 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(2, node2-scads05, 34516)
15/08/02 13:26:44 INFO storage.BlockManagerMaster: Removed 2 successfully in removeExecutor
15/08/02 13:26:49 INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:26:49 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:26:49 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000004 for on host node3-scads06
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000005 for on host node2-scads05
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:26:54 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:26:54 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000005/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000005/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 4, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000004/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000004/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:26:54 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 3, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:26:54 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:26:56 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:35705/user/Executor#1553649062] with ID 4
15/08/02 13:26:56 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 0.0 (TID 2, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:26:56 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:57874/user/Executor#723963404] with ID 3
15/08/02 13:26:56 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 0.0 (TID 3, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:26:56 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:40130 with 3.3 GB RAM, BlockManagerId(4, node2-scads05, 40130)
15/08/02 13:26:56 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:42588 with 3.3 GB RAM, BlockManagerId(3, node3-scads06, 42588)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:40130 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:42588 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:42588 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:26:56 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:40130 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:26:58 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:40130 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:26:59 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:42588 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:27:57 ERROR cluster.YarnClusterScheduler: Lost executor 4 on node2-scads05: remote Akka client disassociated
15/08/02 13:27:57 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:35705] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:27:57 INFO scheduler.TaskSetManager: Re-queueing tasks for 4 from TaskSet 0.0
15/08/02 13:27:57 WARN scheduler.TaskSetManager: Lost task 1.1 in stage 0.0 (TID 2, node2-scads05): ExecutorLostFailure (executor 4 lost)
15/08/02 13:27:57 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 4
15/08/02 13:27:57 INFO scheduler.DAGScheduler: Executor lost: 4 (epoch 2)
15/08/02 13:27:57 INFO storage.BlockManagerMasterActor: Trying to remove executor 4 from BlockManagerMaster.
15/08/02 13:27:57 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(4, node2-scads05, 40130)
15/08/02 13:27:57 INFO storage.BlockManagerMaster: Removed 4 successfully in removeExecutor
15/08/02 13:27:59 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000005 (state: COMPLETE, exit status: -104)
15/08/02 13:27:59 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:28:01 ERROR cluster.YarnClusterScheduler: Lost executor 3 on node3-scads06: remote Akka client disassociated
15/08/02 13:28:01 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:57874] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:28:01 INFO scheduler.TaskSetManager: Re-queueing tasks for 3 from TaskSet 0.0
15/08/02 13:28:01 WARN scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID 3, node3-scads06): ExecutorLostFailure (executor 3 lost)
15/08/02 13:28:01 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 3
15/08/02 13:28:01 INFO scheduler.DAGScheduler: Executor lost: 3 (epoch 3)
15/08/02 13:28:01 INFO storage.BlockManagerMasterActor: Trying to remove executor 3 from BlockManagerMaster.
15/08/02 13:28:01 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(3, node3-scads06, 42588)
15/08/02 13:28:01 INFO storage.BlockManagerMaster: Removed 3 successfully in removeExecutor
15/08/02 13:28:04 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:28:04 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:28:04 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000004 (state: COMPLETE, exit status: -104)
15/08/02 13:28:04 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:28:09 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:28:09 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:28:09 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000006 for on host node3-scads06
15/08/02 13:28:09 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:28:09 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
15/08/02 13:28:09 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000006/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000006/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:28:09 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 5, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:28:09 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:28:11 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:34403/user/Executor#312841975] with ID 5
15/08/02 13:28:11 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 0.0 (TID 4, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:28:11 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:58302 with 3.3 GB RAM, BlockManagerId(5, node3-scads06, 58302)
15/08/02 13:28:11 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:58302 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:28:11 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:58302 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:28:14 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node3-scads06:58302 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:28:14 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000007 for on host node2-scads05
15/08/02 13:28:14 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:28:14 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 1 of them.
15/08/02 13:28:14 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000007/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000007/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:28:14 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 6, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:28:14 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:28:16 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:44899/user/Executor#112632092] with ID 6
15/08/02 13:28:16 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 0.0 (TID 5, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:28:16 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:40856 with 3.3 GB RAM, BlockManagerId(6, node2-scads05, 40856)
15/08/02 13:28:16 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:40856 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:28:16 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:40856 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:28:18 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:40856 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:29:10 ERROR cluster.YarnClusterScheduler: Lost executor 5 on node3-scads06: remote Akka client disassociated
15/08/02 13:29:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:34403] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:29:10 INFO scheduler.TaskSetManager: Re-queueing tasks for 5 from TaskSet 0.0
15/08/02 13:29:10 WARN scheduler.TaskSetManager: Lost task 0.2 in stage 0.0 (TID 4, node3-scads06): ExecutorLostFailure (executor 5 lost)
15/08/02 13:29:10 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 5
15/08/02 13:29:10 INFO scheduler.DAGScheduler: Executor lost: 5 (epoch 4)
15/08/02 13:29:10 INFO storage.BlockManagerMasterActor: Trying to remove executor 5 from BlockManagerMaster.
15/08/02 13:29:10 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(5, node3-scads06, 58302)
15/08/02 13:29:10 INFO storage.BlockManagerMaster: Removed 5 successfully in removeExecutor
15/08/02 13:29:14 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000006 (state: COMPLETE, exit status: -104)
15/08/02 13:29:14 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.1 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:29:18 ERROR cluster.YarnClusterScheduler: Lost executor 6 on node2-scads05: remote Akka client disassociated
15/08/02 13:29:18 INFO scheduler.TaskSetManager: Re-queueing tasks for 6 from TaskSet 0.0
15/08/02 13:29:18 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node2-scads05:44899] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:29:18 WARN scheduler.TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5, node2-scads05): ExecutorLostFailure (executor 6 lost)
15/08/02 13:29:18 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 6
15/08/02 13:29:18 INFO scheduler.DAGScheduler: Executor lost: 6 (epoch 5)
15/08/02 13:29:18 INFO storage.BlockManagerMasterActor: Trying to remove executor 6 from BlockManagerMaster.
15/08/02 13:29:18 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(6, node2-scads05, 40856)
15/08/02 13:29:18 INFO storage.BlockManagerMaster: Removed 6 successfully in removeExecutor
15/08/02 13:29:19 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:29:19 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:29:19 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000007 (state: COMPLETE, exit status: -104)
15/08/02 13:29:19 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:29:24 INFO yarn.YarnAllocator: Will request 1 executor containers, each with 1 cores and 6938 MB memory including 453 MB overhead
15/08/02 13:29:24 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:6938, vCores:1>)
15/08/02 13:29:24 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000009 for on host node3-scads06
15/08/02 13:29:24 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node3-scads06
15/08/02 13:29:24 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:29:24 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000009/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node3-scads06:8042/node/containerlogs/container_1438417242433_0206_01_000009/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:29:24 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 7, --hostname, node3-scads06, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:29:24 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node3-scads06:8041
15/08/02 13:29:26 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node3-scads06:49082/user/Executor#624674070] with ID 7
15/08/02 13:29:26 INFO scheduler.TaskSetManager: Starting task 1.3 in stage 0.0 (TID 6, node3-scads06, NODE_LOCAL, 1316 bytes)
15/08/02 13:29:26 INFO storage.BlockManagerMasterActor: Registering block manager node3-scads06:36761 with 3.3 GB RAM, BlockManagerId(7, node3-scads06, 36761)
15/08/02 13:29:26 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node3-scads06:36761 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:29:26 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node3-scads06:36761 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:29:28 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node3-scads06:36761 (size: 18.8 MB, free: 3.3 GB)
15/08/02 13:29:29 INFO yarn.YarnAllocator: Launching container container_1438417242433_0206_01_000010 for on host node2-scads05
15/08/02 13:29:29 INFO yarn.YarnAllocator: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler,  executorHostname: node2-scads05
15/08/02 13:29:29 INFO yarn.YarnAllocator: Received 2 containers from YARN, launching executors on 1 of them.
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Starting Executor Container
15/08/02 13:29:29 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Preparing Local resources
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__app__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar" } size: 119619578 timestamp: 1438514723931 type: FILE visibility: PRIVATE, __spark__.jar -> resource { scheme: "hdfs" host: "node1-scads02" port: 8020 file: "/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar" } size: 159319006 timestamp: 1438514722782 type: FILE visibility: PRIVATE)
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CLIENT_CONF_DIR<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$MR2_CLASSPATH, SPARK_LOG_URL_STDERR -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000010/marcel/stderr?start=0, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1438417242433_0206, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 159319006,119619578, SPARK_USER -> marcel, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1438514722782,1438514723931, SPARK_LOG_URL_STDOUT -> http://node2-scads05:8042/node/containerlogs/container_1438417242433_0206_01_000010/marcel/stdout?start=0, SPARK_YARN_CACHE_FILES -> hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/spark-assembly-1.3.0-hadoop2.4.0.jar#__spark__.jar,hdfs://node1-scads02:8020/user/marcel/.sparkStaging/application_1438417242433_0206/ma-spark.jar#__app__.jar)
15/08/02 13:29:29 INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms6485m, -Xmx6485m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui.port=0', '-Dspark.driver.port=54443', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, akka.tcp://sparkDriver@node2-scads05:54443/user/CoarseGrainedScheduler, --executor-id, 8, --hostname, node2-scads05, --cores, 1, --app-id, application_1438417242433_0206, --user-class-path, file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/08/02 13:29:29 INFO impl.ContainerManagementProtocolProxy: Opening proxy : node2-scads05:8041
15/08/02 13:29:31 INFO cluster.YarnClusterSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@node2-scads05:36737/user/Executor#-1106619214] with ID 8
15/08/02 13:29:31 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 0.0 (TID 7, node2-scads05, NODE_LOCAL, 1316 bytes)
15/08/02 13:29:31 INFO storage.BlockManagerMasterActor: Registering block manager node2-scads05:44866 with 3.3 GB RAM, BlockManagerId(8, node2-scads05, 44866)
15/08/02 13:29:31 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on node2-scads05:44866 (size: 3.2 KB, free: 3.3 GB)
15/08/02 13:29:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on node2-scads05:44866 (size: 25.4 KB, free: 3.3 GB)
15/08/02 13:29:34 INFO storage.BlockManagerInfo: Added rdd_3_0 in memory on node2-scads05:44866 (size: 44.1 MB, free: 3.2 GB)
15/08/02 13:30:28 ERROR cluster.YarnClusterScheduler: Lost executor 7 on node3-scads06: remote Akka client disassociated
15/08/02 13:30:28 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkExecutor@node3-scads06:49082] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
15/08/02 13:30:28 INFO scheduler.TaskSetManager: Re-queueing tasks for 7 from TaskSet 0.0
15/08/02 13:30:28 WARN scheduler.TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6, node3-scads06): ExecutorLostFailure (executor 7 lost)
15/08/02 13:30:28 ERROR scheduler.TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting job
15/08/02 13:30:28 ERROR cluster.YarnClusterSchedulerBackend: Asked to remove non-existent executor 7
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Cancelling stage 0
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Stage 0 was cancelled
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Job 0 failed: collect at SparkBenchmarkLauncher.java:246, took 286.500864 s
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, node3-scads06): ExecutorLostFailure (executor 7 lost)
Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Executor lost: 7 (epoch 6)
15/08/02 13:30:28 INFO storage.BlockManagerMasterActor: Trying to remove executor 7 from BlockManagerMaster.
15/08/02 13:30:28 INFO storage.BlockManagerMasterActor: Removing block manager BlockManagerId(7, node3-scads06, 36761)
15/08/02 13:30:28 INFO storage.BlockManagerMaster: Removed 7 successfully in removeExecutor
15/08/02 13:30:28 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 0.0 (TID 7, node2-scads05): TaskKilled (killed intentionally)
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/08/02 13:30:28 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/08/02 13:30:28 INFO spark.SparkContext: Starting job: saveAsTextFile at SparkBenchmarkLauncher.java:293
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Got job 1 (saveAsTextFile at SparkBenchmarkLauncher.java:293) with 4 output partitions (allowLocal=false)
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Final stage: Stage 2(saveAsTextFile at SparkBenchmarkLauncher.java:293)
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Missing parents: List()
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Submitting Stage 2 (MapPartitionsRDD[9] at saveAsTextFile at SparkBenchmarkLauncher.java:293), which has no missing parents
15/08/02 13:30:28 INFO storage.MemoryStore: ensureFreeSpace(121336) called with curMem=150925, maxMem=257635123
15/08/02 13:30:28 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 118.5 KB, free 245.4 MB)
15/08/02 13:30:28 INFO storage.MemoryStore: ensureFreeSpace(57208) called with curMem=272261, maxMem=257635123
15/08/02 13:30:28 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 55.9 KB, free 245.4 MB)
15/08/02 13:30:28 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on node2-scads05:50154 (size: 55.9 KB, free: 245.6 MB)
15/08/02 13:30:28 INFO storage.BlockManagerMaster: Updated info of block broadcast_2_piece0
15/08/02 13:30:28 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:839
15/08/02 13:30:28 INFO scheduler.DAGScheduler: Submitting 4 missing tasks from Stage 2 (MapPartitionsRDD[9] at saveAsTextFile at SparkBenchmarkLauncher.java:293)
15/08/02 13:30:28 INFO cluster.YarnClusterScheduler: Adding task set 2.0 with 4 tasks
15/08/02 13:30:28 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 8, node2-scads05, PROCESS_LOCAL, 1327 bytes)
15/08/02 13:30:28 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on node2-scads05:44866 (size: 55.9 KB, free: 3.2 GB)
15/08/02 13:30:29 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 2.0 (TID 9, node2-scads05, NODE_LOCAL, 1327 bytes)
15/08/02 13:30:29 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 8) in 690 ms on node2-scads05 (1/4)
15/08/02 13:30:29 INFO yarn.YarnAllocator: Completed container container_1438417242433_0206_01_000009 (state: COMPLETE, exit status: -104)
15/08/02 13:30:29 WARN yarn.YarnAllocator: Container killed by YARN for exceeding memory limits. 7.3 GB of 7 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
15/08/02 13:30:30 INFO storage.BlockManagerInfo: Added rdd_3_1 in memory on node2-scads05:44866 (size: 18.8 MB, free: 3.2 GB)
15/08/02 13:30:30 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 2.0 (TID 10, node2-scads05, NODE_LOCAL, 1327 bytes)
15/08/02 13:30:30 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 2.0 (TID 9) in 926 ms on node2-scads05 (2/4)
15/08/02 13:30:32 INFO storage.BlockManagerInfo: Added rdd_3_2 in memory on node2-scads05:44866 (size: 43.8 MB, free: 3.2 GB)
15/08/02 13:30:32 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 2.0 (TID 11, node2-scads05, NODE_LOCAL, 1327 bytes)
15/08/02 13:30:32 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 2.0 (TID 10) in 1835 ms on node2-scads05 (3/4)
15/08/02 13:30:33 INFO storage.BlockManagerInfo: Added rdd_3_3 in memory on node2-scads05:44866 (size: 19.2 MB, free: 3.2 GB)
15/08/02 13:30:33 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 2.0 (TID 11) in 885 ms on node2-scads05 (4/4)
15/08/02 13:30:33 INFO cluster.YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool
15/08/02 13:30:33 INFO scheduler.DAGScheduler: Stage 2 (saveAsTextFile at SparkBenchmarkLauncher.java:293) finished in 4.336 s
15/08/02 13:30:33 INFO scheduler.DAGScheduler: Job 1 finished: saveAsTextFile at SparkBenchmarkLauncher.java:293, took 4.379196 s
15/08/02 13:30:33 ERROR yarn.ApplicationMaster: User class threw exception: File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

java.io.FileNotFoundException: File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1144)
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1132)
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1122)
	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1295)
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
	at mgm.tp.bigdata.ma_spark.SparkBenchmarkLauncher.analyseIterationLogandDeleteThem(SparkBenchmarkLauncher.java:147)
	at mgm.tp.bigdata.ma_spark.SparkBenchmarkLauncher.main(SparkBenchmarkLauncher.java:132)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

	at org.apache.hadoop.ipc.Client.call(Client.java:1410)
	at org.apache.hadoop.ipc.Client.call(Client.java:1363)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
	at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1142)
	... 18 more
15/08/02 13:30:33 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: File does not exist: /user/marcel/outputs/output_spark/log0
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
)
15/08/02 13:30:33 INFO yarn.ApplicationMaster: Invoking sc stop from shutdown hook
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
15/08/02 13:30:33 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
15/08/02 13:30:33 INFO ui.SparkUI: Stopped Spark web UI at http://node2-scads05:41412
15/08/02 13:30:33 INFO scheduler.DAGScheduler: Stopping DAGScheduler
15/08/02 13:30:33 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
15/08/02 13:30:33 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
15/08/02 13:30:33 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
15/08/02 13:30:33 INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
15/08/02 13:30:33 INFO storage.MemoryStore: MemoryStore cleared
15/08/02 13:30:33 INFO storage.BlockManager: BlockManager stopped
15/08/02 13:30:33 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
15/08/02 13:30:33 INFO spark.SparkContext: Successfully stopped SparkContext



Am 02.08.2015 um 16:23 schrieb Connor Zanin:
>
> I agree with Ted. Could you please post the log file?
>
> On Aug 2, 2015 10:13 AM, "Ted Yu" <yuzhihong@gmail.com 
> <mailto:yuzhihong@gmail.com>> wrote:
>
>     Can you provide some more detai:
>
>     release of Spark you're using
>     were you running in standalone or YARN cluster mode
>     have you checked driver log ?
>
>     Cheers
>
>     On Sun, Aug 2, 2015 at 7:04 AM, Pa Rö
>     <paul.roewer1990@googlemail.com
>     <mailto:paul.roewer1990@googlemail.com>> wrote:
>
>         hi community,
>
>         i have run my k-means spark application on 1million data
>         points. the program works, but no output in the hdfs is
>         generated. when it runs on 10.000 points, a output is written.
>
>         maybe someone has an idea?
>
>         best regards,
>         paul
>
>


Mime
View raw message