spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pa Rö <paul.roewer1...@googlemail.com>
Subject Re: Asked to remove non-existent executor exception
Date Sun, 26 Jul 2015 07:43:21 GMT
 i has seen that the "tempfs" is full, how i can clear this?

2015-07-23 13:41 GMT+02:00 Pa Rö <paul.roewer1990@googlemail.com>:

> hello spark community,
>
> i have build an application with geomesa, accumulo and spark.
> if it run on spark local mode, it is working, but on spark
> cluster not. in short it says: No space left on device. Asked to remove
> non-existent executor XY.
> I´m confused, because there were many GB´s of free space. do i need to
> change my configuration or what else can i do? thanks in advance.
>
> here is the complete exception:
>
> og4j:WARN No appenders could be found for logger
> (org.apache.accumulo.fate.zookeeper.ZooSession).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 15/07/23 13:26:39 INFO SparkContext: Running Spark version 1.3.0
> 15/07/23 13:26:39 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/07/23 13:26:39 INFO SecurityManager: Changing view acls to: marcel
> 15/07/23 13:26:39 INFO SecurityManager: Changing modify acls to: marcel
> 15/07/23 13:26:39 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(marcel); users
> with modify permissions: Set(marcel)
> 15/07/23 13:26:39 INFO Slf4jLogger: Slf4jLogger started
> 15/07/23 13:26:40 INFO Remoting: Starting remoting
> 15/07/23 13:26:40 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@node1-scads02:52478]
> 15/07/23 13:26:40 INFO Utils: Successfully started service 'sparkDriver'
> on port 52478.
> 15/07/23 13:26:40 INFO SparkEnv: Registering MapOutputTracker
> 15/07/23 13:26:40 INFO SparkEnv: Registering BlockManagerMaster
> 15/07/23 13:26:40 INFO DiskBlockManager: Created local directory at
> /tmp/spark-ca9319d4-68a2-4add-a21a-48b13ae9cf81/blockmgr-cbf8af23-e113-4732-8c2c-7413ad237b3b
> 15/07/23 13:26:40 INFO MemoryStore: MemoryStore started with capacity
> 1916.2 MB
> 15/07/23 13:26:40 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-9d4a04d5-3535-49e0-a859-d278a0cc7bf8/httpd-1882aafc-45fe-4490-803d-c04fc67510a2
> 15/07/23 13:26:40 INFO HttpServer: Starting HTTP Server
> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT
> 15/07/23 13:26:40 INFO AbstractConnector: Started
> SocketConnector@0.0.0.0:56499
> 15/07/23 13:26:40 INFO Utils: Successfully started service 'HTTP file
> server' on port 56499.
> 15/07/23 13:26:40 INFO SparkEnv: Registering OutputCommitCoordinator
> 15/07/23 13:26:40 INFO Server: jetty-8.y.z-SNAPSHOT
> 15/07/23 13:26:40 INFO AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
> 15/07/23 13:26:40 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
> 15/07/23 13:26:40 INFO SparkUI: Started SparkUI at
> http://node1-scads02:4040
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@node1-scads02:7077/user/Master...
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Connected to Spark
> cluster with app ID app-20150723132640-0000
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
> app-20150723132640-0000/0 on worker-20150723132524-node3-scads06-7078
> (node3-scads06:7078) with 8 cores
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20150723132640-0000/0 on hostPort node3-scads06:7078 with 8 cores,
> 512.0 MB RAM
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
> app-20150723132640-0000/1 on worker-20150723132513-node2-scads05-7078
> (node2-scads05:7078) with 8 cores
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20150723132640-0000/1 on hostPort node2-scads05:7078 with 8 cores,
> 512.0 MB RAM
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/0 is now RUNNING
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/1 is now RUNNING
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/0 is now LOADING
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/1 is now LOADING
> 15/07/23 13:26:40 INFO NettyBlockTransferService: Server created on 45786
> 15/07/23 13:26:40 INFO BlockManagerMaster: Trying to register BlockManager
> 15/07/23 13:26:40 INFO BlockManagerMasterActor: Registering block manager
> node1-scads02:45786 with 1916.2 MB RAM, BlockManagerId(<driver>,
> node1-scads02, 45786)
> 15/07/23 13:26:40 INFO BlockManagerMaster: Registered BlockManager
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/0 is now FAILED (java.io.IOException: No space left
> on device)
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor
> app-20150723132640-0000/0 removed: java.io.IOException: No space left on
> device
> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove
> non-existent executor 0
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
> app-20150723132640-0000/2 on worker-20150723132524-node3-scads06-7078
> (node3-scads06:7078) with 8 cores
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20150723132640-0000/2 on hostPort node3-scads06:7078 with 8 cores,
> 512.0 MB RAM
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/1 is now FAILED (java.io.IOException: No space left
> on device)
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Executor
> app-20150723132640-0000/1 removed: java.io.IOException: No space left on
> device
> 15/07/23 13:26:40 ERROR SparkDeploySchedulerBackend: Asked to remove
> non-existent executor 1
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor added:
> app-20150723132640-0000/3 on worker-20150723132513-node2-scads05-7078
> (node2-scads05:7078) with 8 cores
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: Granted executor ID
> app-20150723132640-0000/3 on hostPort node2-scads05:7078 with 8 cores,
> 512.0 MB RAM
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/2 is now LOADING
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/3 is now LOADING
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/2 is now RUNNING
> 15/07/23 13:26:40 INFO SparkDeploySchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.0
> 15/07/23 13:26:40 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/3 is now RUNNING
> 15/07/23 13:26:41 INFO AppClient$ClientActor: Executor updated:
> app-20150723132640-0000/2 is now FAILED (java.io.IOException: No space left
> on device)
> 15/07/23 13:26:41 INFO SparkDeploySchedulerBackend: Executor
> app-20150723132640-0000/2 removed: java.io.IOException: No space left on
> device
> 15/07/23 13:26:41 ERROR SparkDeploySchedulerBackend: Asked to remove
> non-existent executor 2...
>

Mime
View raw message