spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <luohui20...@sina.com>
Subject 回复:Re: How to decrease the time of storing block in memory
Date Tue, 09 Jun 2015 08:39:26 GMT
Only 1 minor GC, 0.07s. 


--------------------------------

 

Thanks&amp;Best regards!
San.Luo

----- 原始邮件 -----
发件人:Akhil Das <akhil@sigmoidanalytics.com>
收件人:罗辉 <luohui20001@sina.com>
抄送人:user <user@spark.apache.org>
主题:Re: How to decrease the time of storing block in memory
日期:2015年06月09日 15点02分

May be you should check in your driver UI and see if there's any GC time involved etc. ThanksBest
Regards

On Mon, Jun 8, 2015 at 5:45 PM,  <luohui20001@sina.com> wrote:
hi there      I am trying to descrease my app's running time in worker node. I checked the
log and found the most time-wasting part is below:15/06/08 16:14:23 INFO storage.MemoryStore:
Block broadcast_0 stored as values in memory (estimated size 2.1 KB, free 353.3 MB)
15/06/08 16:14:42 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 693 bytes
result sent to driver

I don't know why it needs 19s to storing 2.1KB size data to memory. Is there any tuning method?

The attache is the full log, here it is:15/06/08 16:14:02 INFO executor.CoarseGrainedExecutorBackend:
Registered signal handlers for [TERM, HUP, INT]
15/06/08 16:14:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
15/06/08 16:14:10 INFO spark.SecurityManager: Changing view acls to: root
15/06/08 16:14:10 INFO spark.SecurityManager: Changing modify acls to: root
15/06/08 16:14:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/06/08 16:14:14 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/06/08 16:14:14 INFO Remoting: Starting remoting
15/06/08 16:14:15 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://driverPropsFetcher@slave5:54684]
15/06/08 16:14:15 INFO util.Utils: Successfully started service 'driverPropsFetcher' on port
54684.
15/06/08 16:14:16 INFO spark.SecurityManager: Changing view acls to: root
15/06/08 16:14:16 INFO spark.SecurityManager: Changing modify acls to: root
15/06/08 16:14:16 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/06/08 16:14:16 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote
daemon.
15/06/08 16:14:16 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut
down; proceeding with flushing remote transports.
15/06/08 16:14:16 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/06/08 16:14:16 INFO Remoting: Starting remoting
15/06/08 16:14:17 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
15/06/08 16:14:17 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@slave5:49169]
15/06/08 16:14:17 INFO util.Utils: Successfully started service 'sparkExecutor' on port 49169.
15/06/08 16:14:17 INFO util.AkkaUtils: Connecting to MapOutputTracker: akka.tcp://sparkDriver@slave5:58630/user/MapOutputTracker
15/06/08 16:14:17 INFO util.AkkaUtils: Connecting to BlockManagerMaster: akka.tcp://sparkDriver@slave5:58630/user/BlockManagerMaster
15/06/08 16:14:17 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-548b4618-4aba-4b63-9467-381fbfea8d5b/spark-83737dd6-46b0-47ee-82a5-5afee46bdbf5/spark-0fe9d8ba-2910-44a2-bf4f-80d179f5d58b/blockmgr-b4884e7a-2527-447a-9fc5-1823d923c2f1
15/06/08 16:14:17 INFO storage.MemoryStore: MemoryStore started with capacity 353.3 MB
15/06/08 16:14:17 INFO util.AkkaUtils: Connecting to OutputCommitCoordinator: akka.tcp://sparkDriver@slave5:58630/user/OutputCommitCoordinator
15/06/08 16:14:17 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sparkDriver@slave5:58630/user/CoarseGrainedScheduler
15/06/08 16:14:18 INFO worker.WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@slave5:48926/user/Worker
15/06/08 16:14:18 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with
driver
15/06/08 16:14:18 INFO executor.Executor: Starting executor ID 0 on host slave5
15/06/08 16:14:18 INFO worker.WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@slave5:48926/user/Worker
15/06/08 16:14:21 WARN internal.ThreadLocalRandom: Failed to generate a seed from SecureRandom
within 3 seconds. Not enough entrophy?
15/06/08 16:14:21 INFO netty.NettyBlockTransferService: Server created on 53449
15/06/08 16:14:21 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/06/08 16:14:21 INFO storage.BlockManagerMaster: Registered BlockManager
15/06/08 16:14:21 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@slave5:58630/user/HeartbeatReceiver
15/06/08 16:14:21 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 0
15/06/08 16:14:21 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
15/06/08 16:14:21 INFO executor.Executor: Fetching http://192.168.100.11:50648/jars/ShellCompare.jar
with timestamp 1433751228699
15/06/08 16:14:21 INFO util.Utils: Fetching http://192.168.100.11:50648/jars/ShellCompare.jar
to /tmp/spark-548b4618-4aba-4b63-9467-381fbfea8d5b/spark-83737dd6-46b0-47ee-82a5-5afee46bdbf5/spark-99aba156-5803-41b8-9df1-3e36305f43c3/fetchFileTemp8512004298922421624.tmp
15/06/08 16:14:21 INFO util.Utils: Copying /tmp/spark-548b4618-4aba-4b63-9467-381fbfea8d5b/spark-83737dd6-46b0-47ee-82a5-5afee46bdbf5/spark-99aba156-5803-41b8-9df1-3e36305f43c3/-8058314591433751228699_cache
to /usr/lib/spark/work/app-20150608161350-0001/0/./ShellCompare.jar
15/06/08 16:14:22 INFO executor.Executor: Adding file:/usr/lib/spark/work/app-20150608161350-0001/0/./ShellCompare.jar
to class loader
15/06/08 16:14:22 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 0
15/06/08 16:14:23 INFO storage.MemoryStore: ensureFreeSpace(1561) called with curMem=0, maxMem=370503843
15/06/08 16:14:23 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory
(estimated size 1561.0 B, free 353.3 MB)
15/06/08 16:14:23 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
15/06/08 16:14:23 INFO broadcast.TorrentBroadcast: Reading broadcast variable 0 took 967 ms
15/06/08 16:14:23 INFO storage.MemoryStore: ensureFreeSpace(2168) called with curMem=1561,
maxMem=370503843
15/06/08 16:14:23 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated
size 2.1 KB, free 353.3 MB)
15/06/08 16:14:42 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 693 bytes
result sent to driver
15/06/08 16:14:42 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 1
15/06/08 16:14:42 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1)
15/06/08 16:14:56 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 693 bytes
result sent to driver
15/06/08 16:14:56 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown
15/06/08 16:14:56 INFO storage.MemoryStore: MemoryStore cleared
15/06/08 16:14:56 INFO storage.BlockManager: BlockManager stopped
15/06/08 16:14:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote
daemon.
15/06/08 16:14:56 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut
down; proceeding with flushing remote transports.


--------------------------------

 

Thanks&amp;Best regards!
San.Luo



---------------------------------------------------------------------

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org

For additional commands, e-mail: user-help@spark.apache.org



Mime
View raw message