spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haiyang Fu <haiyangfu...@gmail.com>
Subject Re: java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
Date Thu, 31 Jul 2014 12:05:34 GMT
Have you tried to increase the dirver memory?


On Thu, Jul 31, 2014 at 3:54 PM, Bin <wubin_phight@126.com> wrote:

> Hi All,
>
> The data size of my task is about 30mb. It runs smoothly in local mode.
> However, when I submit it to the cluster, it throws the titled error
> (Please see below for the complete output).
>
> Actually, my output is almost the same with
> http://stackoverflow.com/questions/24080891/spark-program-hangs-at-job-finished-toarray-workers-throw-java-util-concurren.
I
> also toArray my data, which was the reason of his case.
>
> However, how come it runs OK in local but not in the cluster? The memory
> of each worker is over 60g, and my run command is:
>
> "$SPARK_HOME/bin/spark-class org.apache.spark.deploy.Client launch spark://
> 10.196.135.101:7077 $jar_path $programname -Dspark.master=spark://
> 10.196.135.101:7077 -Dspark.cores.max=300 -Dspark.executor.memory=20g
> -spark.jars=$jar_path -Dspark.default.parallelism=100
>  -Dspark.hadoop.hadoop.job.ugi=$username,$groupname  -Dspark.app.name=$appname
> $in_path $scala_out_path"
>
> Looking for help and thanks a lot!
>
> Below please find the complete output:
>
> 14/07/31 15:06:53 WARN Configuration: DEPRECATED: hadoop-site.xml found in the classpath.
Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml
to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
> 14/07/31 15:06:53 INFO SecurityManager: Changing view acls to: spark
> 14/07/31 15:06:53 INFO SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(spark)
> 14/07/31 15:06:53 INFO Slf4jLogger: Slf4jLogger started
> 14/07/31 15:06:53 INFO Remoting: Starting remoting
> 14/07/31 15:06:54 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor@tdw-10-215-140-22:39446]
> 14/07/31 15:06:54 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkExecutor@tdw-10-215-140-22:39446]
> 14/07/31 15:06:54 INFO CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://spark@tdw-10-196-135-106:38502/user/CoarseGrainedScheduler
> 14/07/31 15:06:54 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@tdw-10-215-140-22:34755/user/Worker
> 14/07/31 15:06:54 INFO WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@tdw-10-215-140-22:34755/user/Worker
> 14/07/31 15:06:56 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
> 14/07/31 15:06:56 INFO SecurityManager: Changing view acls to: spark
> 14/07/31 15:06:56 INFO SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(spark)
> 14/07/31 15:06:56 INFO Slf4jLogger: Slf4jLogger started
> 14/07/31 15:06:56 INFO Remoting: Starting remoting
> 14/07/31 15:06:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@tdw-10-215-140-22:56708]
> 14/07/31 15:06:56 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@tdw-10-215-140-22:56708]
> 14/07/31 15:06:56 INFO SparkEnv: Connecting to MapOutputTracker: akka.tcp://spark@tdw-10-196-135-106:38502/user/MapOutputTracker
> 14/07/31 15:06:58 INFO SparkEnv: Connecting to BlockManagerMaster: akka.tcp://spark@tdw-10-196-135-106:38502/user/BlockManagerMaster
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data1/sparkenv/local/spark-local-20140731150659-3f12
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data2/sparkenv/local/spark-local-20140731150659-1602
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data3/sparkenv/local/spark-local-20140731150659-d213
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data4/sparkenv/local/spark-local-20140731150659-f42e
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data5/sparkenv/local/spark-local-20140731150659-63d0
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data6/sparkenv/local/spark-local-20140731150659-9003
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data7/sparkenv/local/spark-local-20140731150659-f260
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data8/sparkenv/local/spark-local-20140731150659-6334
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data9/sparkenv/local/spark-local-20140731150659-3af4
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data10/sparkenv/local/spark-local-20140731150659-133d
> 14/07/31 15:06:59 INFO DiskBlockManager: Created local directory at /data11/sparkenv/local/spark-local-20140731150659-ed08
> 14/07/31 15:06:59 INFO MemoryStore: MemoryStore started with capacity 11.5 GB.
> 14/07/31 15:06:59 INFO ConnectionManager: Bound socket to port 35127 with id = ConnectionManagerId(tdw-10-215-140-22,35127)
> 14/07/31 15:06:59 INFO BlockManagerMaster: Trying to register BlockManager
> 14/07/31 15:07:00 INFO BlockManagerMaster: Registered BlockManager
> 14/07/31 15:07:00 INFO HttpFileServer: HTTP File server directory is /tmp/spark-0914d215-dd22-4d5e-9ec0-724937dbfd8b
> 14/07/31 15:07:00 INFO HttpServer: Starting HTTP Server
> 14/07/31 15:07:26 INFO CoarseGrainedExecutorBackend: Got assigned task 12
> 14/07/31 15:07:26 INFO CoarseGrainedExecutorBackend: Got assigned task 25
> 14/07/31 15:07:26 INFO Executor: Running task ID 25
> 14/07/31 15:07:26 INFO Executor: Running task ID 12
> 14/07/31 15:07:26 INFO Executor: Fetching hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/adsorption_2.10-1.0.jar
with timestamp 1406790410442
> 14/07/31 15:07:26 INFO Executor: Adding file:/data/home/spark/spark-1.0.0-bin-0.20.2-cdh3u3/work/app-20140731150652-4911/8/./adsorption_2.10-1.0.jar
to class loader
> 14/07/31 15:07:26 INFO HttpBroadcast: Started reading broadcast variable 0
> 14/07/31 15:07:28 INFO MemoryStore: ensureFreeSpace(102650) called with curMem=0, maxMem=12348240691
> 14/07/31 15:07:28 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated
size 100.2 KB, free 11.5 GB)
> 14/07/31 15:07:28 INFO HttpBroadcast: Reading broadcast variable 0 took 1.702291406 s
> 14/07/31 15:07:28 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:28 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00072:0+197955
> 14/07/31 15:07:28 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00062:0+218630
> 14/07/31 15:07:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
> 14/07/31 15:07:28 WARN LoadSnappy: Snappy native library not loaded
> 14/07/31 15:07:29 INFO Executor: Serialized size of result for 12 is 887
> 14/07/31 15:07:29 INFO Executor: Serialized size of result for 25 is 887
> 14/07/31 15:07:29 INFO Executor: Sending result for 25 directly to driver
> 14/07/31 15:07:29 INFO Executor: Sending result for 12 directly to driver
> 14/07/31 15:07:29 INFO Executor: Finished task ID 12
> 14/07/31 15:07:29 INFO Executor: Finished task ID 25
> 14/07/31 15:07:30 INFO CoarseGrainedExecutorBackend: Got assigned task 30
> 14/07/31 15:07:30 INFO Executor: Running task ID 30
> 14/07/31 15:07:30 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:30 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00084:0+196433
> 14/07/31 15:07:30 INFO Executor: Serialized size of result for 30 is 887
> 14/07/31 15:07:30 INFO Executor: Sending result for 30 directly to driver
> 14/07/31 15:07:30 INFO Executor: Finished task ID 30
> 14/07/31 15:07:30 INFO CoarseGrainedExecutorBackend: Got assigned task 31
> 14/07/31 15:07:30 INFO Executor: Running task ID 31
> 14/07/31 15:07:30 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:30 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00089:0+190194
> 14/07/31 15:07:30 INFO Executor: Serialized size of result for 31 is 887
> 14/07/31 15:07:30 INFO Executor: Sending result for 31 directly to driver
> 14/07/31 15:07:30 INFO Executor: Finished task ID 31
> 14/07/31 15:07:31 INFO CoarseGrainedExecutorBackend: Got assigned task 54
> 14/07/31 15:07:31 INFO Executor: Running task ID 54
> 14/07/31 15:07:31 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:31 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00096:0+153443
> 14/07/31 15:07:31 INFO CoarseGrainedExecutorBackend: Got assigned task 55
> 14/07/31 15:07:31 INFO Executor: Running task ID 55
> 14/07/31 15:07:31 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:31 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00139:0+174726
> 14/07/31 15:07:31 INFO Executor: Serialized size of result for 54 is 887
> 14/07/31 15:07:31 INFO Executor: Sending result for 54 directly to driver
> 14/07/31 15:07:31 INFO Executor: Finished task ID 54
> 14/07/31 15:07:31 INFO Executor: Serialized size of result for 55 is 887
> 14/07/31 15:07:31 INFO Executor: Sending result for 55 directly to driver
> 14/07/31 15:07:31 INFO Executor: Finished task ID 55
> 14/07/31 15:07:32 INFO CoarseGrainedExecutorBackend: Got assigned task 76
> 14/07/31 15:07:32 INFO Executor: Running task ID 76
> 14/07/31 15:07:32 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:32 INFO CoarseGrainedExecutorBackend: Got assigned task 79
> 14/07/31 15:07:32 INFO Executor: Running task ID 79
> 14/07/31 15:07:32 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:32 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00149:0+134758
> 14/07/31 15:07:32 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00157:0+103176
> 14/07/31 15:07:32 INFO Executor: Serialized size of result for 79 is 887
> 14/07/31 15:07:32 INFO Executor: Sending result for 79 directly to driver
> 14/07/31 15:07:32 INFO Executor: Finished task ID 79
> 14/07/31 15:07:32 INFO Executor: Serialized size of result for 76 is 887
> 14/07/31 15:07:32 INFO Executor: Sending result for 76 directly to driver
> 14/07/31 15:07:32 INFO Executor: Finished task ID 76
> 14/07/31 15:07:34 INFO CoarseGrainedExecutorBackend: Got assigned task 99
> 14/07/31 15:07:34 INFO Executor: Running task ID 99
> 14/07/31 15:07:34 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:34 INFO HadoopRDD: Input split: hdfs://tdw-10-196-135-101:54310/user/teg/gdt/tj/test/pre/scala_out/part-00167:0+51005
> 14/07/31 15:07:34 INFO Executor: Serialized size of result for 99 is 887
> 14/07/31 15:07:34 INFO Executor: Sending result for 99 directly to driver
> 14/07/31 15:07:34 INFO Executor: Finished task ID 99
> 14/07/31 15:07:39 INFO CoarseGrainedExecutorBackend: Got assigned task 181
> 14/07/31 15:07:39 INFO Executor: Running task ID 181
> 14/07/31 15:07:39 INFO CoarseGrainedExecutorBackend: Got assigned task 196
> 14/07/31 15:07:39 INFO Executor: Running task ID 196
> 14/07/31 15:07:39 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:39 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-24/10.215.140.24]
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-196-135-105/10.196.135.105]
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-21/10.215.140.21]
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-12/10.215.140.12]
> 46.947: [GC 10486272K->32824K(40196096K), 0.0347340 secs]
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-13/10.215.140.13]
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-23/10.215.140.23]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-13/10.215.140.13:58657]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-23/10.215.140.23:39188]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-21/10.215.140.21:36128]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-12/10.215.140.12:33380]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-196-135-105/10.196.135.105:36859]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-24/10.215.140.24:49100]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-23/10.215.140.23:39188],
2 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-13/10.215.140.13:58657],
2 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-196-135-105/10.196.135.105:36859],
2 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-12/10.215.140.12:33380],
1 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-21/10.215.140.21:36128],
1 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-24/10.215.140.24:49100],
2 messages pending
> 14/07/31 15:07:39 INFO CacheManager: Partition rdd_6_10 not found, computing it
> 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0,
fetching them
> 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Doing the fetch; tracker actor = Actor[akka.tcp://spark@tdw-10-196-135-106:38502/user/MapOutputTracker#-128956169]
> 14/07/31 15:07:39 INFO CacheManager: Partition rdd_6_25 not found, computing it
> 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0,
fetching them
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-17/10.215.140.17]
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-25/10.215.140.25]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-17/10.215.140.17:35040]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-17/10.215.140.17:35040],
1 messages pending
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-18/10.215.140.18]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-25/10.215.140.25:50298]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-25/10.215.140.25:50298],
2 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-18/10.215.140.18:33575]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-18/10.215.140.18:33575],
1 messages pending
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-14/10.215.140.14]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-14/10.215.140.14:37220]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-14/10.215.140.14:37220],
1 messages pending
> 14/07/31 15:07:39 INFO MapOutputTrackerWorker: Got the output locations
> 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 171 non-empty
blocks out of 171 blocks
> 14/07/31 15:07:39 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 171 non-empty
blocks out of 171 blocks
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-196-135-107/10.196.135.107:59290]
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-11/10.215.140.11:50302]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-11/10.215.140.11:50302],
1 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-196-135-106/10.196.135.106:34128]
> 14/07/31 15:07:39 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-11/10.215.140.11]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-196-135-107/10.196.135.107:59290],
2 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-196-135-106/10.196.135.106:34128],
1 messages pending
> 14/07/31 15:07:39 INFO SendingConnection: Initiating connection to [tdw-10-215-140-15/10.215.140.15:50069]
> 14/07/31 15:07:39 INFO SendingConnection: Connected to [tdw-10-215-140-15/10.215.140.15:50069],
1 messages pending
> 14/07/31 15:07:40 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 14 remote
fetches in 48 ms
> 14/07/31 15:07:40 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-15/10.215.140.15]
> 14/07/31 15:07:40 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 14 remote
fetches in 49 ms
> 14/07/31 15:07:40 INFO ConnectionManager: Accepted connection from [tdw-10-215-140-16/10.215.140.16]
> 14/07/31 15:07:40 INFO ConnectionManager: Accepted connection from [tdw-10-196-135-102/10.196.135.102]
> 14/07/31 15:07:40 INFO SendingConnection: Initiating connection to [tdw-10-215-140-16/10.215.140.16:58648]
> 14/07/31 15:07:40 INFO SendingConnection: Connected to [tdw-10-215-140-16/10.215.140.16:58648],
1 messages pending
> 14/07/31 15:07:40 INFO SendingConnection: Initiating connection to [tdw-10-196-135-102/10.196.135.102:45729]
> 14/07/31 15:07:40 INFO SendingConnection: Connected to [tdw-10-196-135-102/10.196.135.102:45729],
1 messages pending
> 14/07/31 15:07:42 INFO ConnectionManager: Accepted connection from [tdw-10-196-135-106/10.196.135.106]
> 14/07/31 15:07:44 INFO ConnectionManager: Accepted connection from [tdw-10-196-135-107/10.196.135.107]
> 14/07/31 15:07:45 INFO MemoryStore: ensureFreeSpace(1922882) called with curMem=102650,
maxMem=12348240691
> 14/07/31 15:07:45 INFO MemoryStore: Block rdd_6_25 stored as values to memory (estimated
size 1877.8 KB, free 11.5 GB)
> 14/07/31 15:07:46 INFO MemoryStore: ensureFreeSpace(1912396) called with curMem=2025532,
maxMem=12348240691
> 14/07/31 15:07:46 INFO MemoryStore: Block rdd_6_10 stored as values to memory (estimated
size 1867.6 KB, free 11.5 GB)
> 14/07/31 15:07:46 INFO BlockManagerMaster: Updated info of block rdd_6_10
> 14/07/31 15:07:46 INFO BlockManagerMaster: Updated info of block rdd_6_25
> 14/07/31 15:07:46 INFO Executor: Serialized size of result for 181 is 421363
> 14/07/31 15:07:46 INFO Executor: Serialized size of result for 196 is 421522
> 14/07/31 15:07:46 INFO Executor: Sending result for 181 directly to driver
> 14/07/31 15:07:46 INFO Executor: Sending result for 196 directly to driver
> 14/07/31 15:07:46 INFO Executor: Finished task ID 181
> 14/07/31 15:07:46 INFO Executor: Finished task ID 196
> 14/07/31 15:07:50 INFO CoarseGrainedExecutorBackend: Got assigned task 219
> 14/07/31 15:07:50 INFO Executor: Running task ID 219
> 14/07/31 15:07:50 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:50 INFO CacheManager: Partition rdd_6_48 not found, computing it
> 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 171 non-empty
blocks out of 171 blocks
> 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 14 remote
fetches in 19 ms
> 14/07/31 15:07:50 INFO CoarseGrainedExecutorBackend: Got assigned task 225
> 14/07/31 15:07:50 INFO Executor: Running task ID 225
> 14/07/31 15:07:50 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:50 INFO CacheManager: Partition rdd_6_54 not found, computing it
> 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 171 non-empty
blocks out of 171 blocks
> 14/07/31 15:07:50 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 14 remote
fetches in 15 ms
> 14/07/31 15:07:51 INFO MemoryStore: ensureFreeSpace(1927469) called with curMem=3937928,
maxMem=12348240691
> 14/07/31 15:07:51 INFO MemoryStore: Block rdd_6_48 stored as values to memory (estimated
size 1882.3 KB, free 11.5 GB)
> 14/07/31 15:07:51 INFO BlockManagerMaster: Updated info of block rdd_6_48
> 14/07/31 15:07:51 INFO Executor: Serialized size of result for 219 is 424342
> 14/07/31 15:07:51 INFO Executor: Sending result for 219 directly to driver
> 14/07/31 15:07:51 INFO Executor: Finished task ID 219
> 14/07/31 15:07:51 INFO MemoryStore: ensureFreeSpace(1909775) called with curMem=5865397,
maxMem=12348240691
> 14/07/31 15:07:51 INFO MemoryStore: Block rdd_6_54 stored as values to memory (estimated
size 1865.0 KB, free 11.5 GB)
> 14/07/31 15:07:51 INFO BlockManagerMaster: Updated info of block rdd_6_54
> 14/07/31 15:07:51 INFO Executor: Serialized size of result for 225 is 421546
> 14/07/31 15:07:51 INFO Executor: Sending result for 225 directly to driver
> 14/07/31 15:07:51 INFO Executor: Finished task ID 225
> 14/07/31 15:07:53 INFO CoarseGrainedExecutorBackend: Got assigned task 251
> 14/07/31 15:07:53 INFO Executor: Running task ID 251
> 14/07/31 15:07:53 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:53 INFO CacheManager: Partition rdd_6_80 not found, computing it
> 14/07/31 15:07:53 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:07:53 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 171 non-empty
blocks out of 171 blocks
> 14/07/31 15:07:53 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 14 remote
fetches in 15 ms
> 14/07/31 15:07:53 INFO MemoryStore: ensureFreeSpace(1927469) called with curMem=7775172,
maxMem=12348240691
> 14/07/31 15:07:53 INFO MemoryStore: Block rdd_6_80 stored as values to memory (estimated
size 1882.3 KB, free 11.5 GB)
> 14/07/31 15:07:53 INFO BlockManagerMaster: Updated info of block rdd_6_80
> 14/07/31 15:07:53 INFO Executor: Serialized size of result for 251 is 424634
> 14/07/31 15:07:53 INFO Executor: Sending result for 251 directly to driver
> 14/07/31 15:07:53 INFO Executor: Finished task ID 251
> 14/07/31 15:07:54 INFO CoarseGrainedExecutorBackend: Got assigned task 259
> 14/07/31 15:07:54 INFO Executor: Running task ID 259
> 14/07/31 15:07:54 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:54 INFO CacheManager: Partition rdd_6_88 not found, computing it
> 14/07/31 15:07:54 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:07:54 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 171 non-empty
blocks out of 171 blocks
> 14/07/31 15:07:54 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 14 remote
fetches in 13 ms
> 14/07/31 15:07:54 INFO MemoryStore: ensureFreeSpace(1921571) called with curMem=9702641,
maxMem=12348240691
> 14/07/31 15:07:54 INFO MemoryStore: Block rdd_6_88 stored as values to memory (estimated
size 1876.5 KB, free 11.5 GB)
> 14/07/31 15:07:54 INFO BlockManagerMaster: Updated info of block rdd_6_88
> 14/07/31 15:07:54 INFO Executor: Serialized size of result for 259 is 418167
> 14/07/31 15:07:54 INFO Executor: Sending result for 259 directly to driver
> 14/07/31 15:07:54 INFO Executor: Finished task ID 259
> 14/07/31 15:07:56 INFO CoarseGrainedExecutorBackend: Got assigned task 273
> 14/07/31 15:07:56 INFO Executor: Running task ID 273
> 14/07/31 15:07:56 INFO CoarseGrainedExecutorBackend: Got assigned task 290
> 14/07/31 15:07:56 INFO Executor: Running task ID 290
> 14/07/31 15:07:56 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:56 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:56 INFO BlockManager: Found block rdd_6_10 locally
> 14/07/31 15:07:56 INFO BlockManager: Found block rdd_6_25 locally
> 14/07/31 15:07:56 INFO Executor: Serialized size of result for 273 is 887
> 14/07/31 15:07:56 INFO Executor: Sending result for 273 directly to driver
> 14/07/31 15:07:56 INFO Executor: Finished task ID 273
> 14/07/31 15:07:56 INFO Executor: Serialized size of result for 290 is 887
> 14/07/31 15:07:56 INFO Executor: Sending result for 290 directly to driver
> 14/07/31 15:07:56 INFO Executor: Finished task ID 290
> 14/07/31 15:07:57 INFO CoarseGrainedExecutorBackend: Got assigned task 308
> 14/07/31 15:07:57 INFO Executor: Running task ID 308
> 14/07/31 15:07:57 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:57 INFO BlockManager: Found block rdd_6_48 locally
> 14/07/31 15:07:57 INFO CoarseGrainedExecutorBackend: Got assigned task 311
> 14/07/31 15:07:57 INFO Executor: Running task ID 311
> 14/07/31 15:07:57 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:57 INFO BlockManager: Found block rdd_6_54 locally
> 14/07/31 15:07:57 INFO Executor: Serialized size of result for 308 is 887
> 14/07/31 15:07:57 INFO Executor: Sending result for 308 directly to driver
> 14/07/31 15:07:57 INFO Executor: Finished task ID 308
> 14/07/31 15:07:57 INFO Executor: Serialized size of result for 311 is 887
> 14/07/31 15:07:57 INFO Executor: Sending result for 311 directly to driver
> 14/07/31 15:07:57 INFO Executor: Finished task ID 311
> 14/07/31 15:07:58 INFO CoarseGrainedExecutorBackend: Got assigned task 339
> 14/07/31 15:07:58 INFO Executor: Running task ID 339
> 14/07/31 15:07:58 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:58 INFO BlockManager: Found block rdd_6_80 locally
> 14/07/31 15:07:58 INFO CoarseGrainedExecutorBackend: Got assigned task 341
> 14/07/31 15:07:58 INFO Executor: Running task ID 341
> 14/07/31 15:07:58 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:58 INFO BlockManager: Found block rdd_6_88 locally
> 14/07/31 15:07:58 INFO Executor: Serialized size of result for 339 is 887
> 14/07/31 15:07:58 INFO Executor: Sending result for 339 directly to driver
> 14/07/31 15:07:58 INFO Executor: Finished task ID 339
> 14/07/31 15:07:58 INFO Executor: Serialized size of result for 341 is 887
> 14/07/31 15:07:58 INFO Executor: Sending result for 341 directly to driver
> 14/07/31 15:07:58 INFO Executor: Finished task ID 341
> 14/07/31 15:07:59 INFO CoarseGrainedExecutorBackend: Got assigned task 377
> 14/07/31 15:07:59 INFO Executor: Running task ID 377
> 14/07/31 15:07:59 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
> 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1,
fetching them
> 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Doing the fetch; tracker actor = Actor[akka.tcp://spark@tdw-10-196-135-106:38502/user/MapOutputTracker#-128956169]
> 14/07/31 15:07:59 INFO MapOutputTrackerWorker: Got the output locations
> 14/07/31 15:07:59 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:07:59 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 100 non-empty
blocks out of 100 blocks
> 14/07/31 15:07:59 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 16 remote
fetches in 9 ms
> 14/07/31 15:08:00 INFO CoarseGrainedExecutorBackend: Got assigned task 393
> 14/07/31 15:08:00 INFO Executor: Running task ID 393
> 14/07/31 15:08:00 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:08:00 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:08:00 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 100 non-empty
blocks out of 100 blocks
> 14/07/31 15:08:00 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 16 remote
fetches in 8 ms
> 14/07/31 15:08:00 INFO Executor: Serialized size of result for 377 is 303256
> 14/07/31 15:08:00 INFO Executor: Sending result for 377 directly to driver
> 14/07/31 15:08:00 INFO Executor: Finished task ID 377
> 14/07/31 15:08:00 INFO Executor: Serialized size of result for 393 is 310660
> 14/07/31 15:08:00 INFO Executor: Sending result for 393 directly to driver
> 14/07/31 15:08:00 INFO Executor: Finished task ID 393
> 14/07/31 15:08:01 INFO CoarseGrainedExecutorBackend: Got assigned task 403
> 14/07/31 15:08:01 INFO Executor: Running task ID 403
> 14/07/31 15:08:01 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:08:01 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:08:01 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 100 non-empty
blocks out of 100 blocks
> 14/07/31 15:08:01 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 16 remote
fetches in 7 ms
> 14/07/31 15:08:01 INFO Executor: Serialized size of result for 403 is 299667
> 14/07/31 15:08:01 INFO Executor: Sending result for 403 directly to driver
> 14/07/31 15:08:01 INFO Executor: Finished task ID 403
> 14/07/31 15:08:02 INFO CoarseGrainedExecutorBackend: Got assigned task 412
> 14/07/31 15:08:02 INFO Executor: Running task ID 412
> 14/07/31 15:08:02 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:08:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:08:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 100 non-empty
blocks out of 100 blocks
> 14/07/31 15:08:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 16 remote
fetches in 6 ms
> 14/07/31 15:08:02 INFO Executor: Serialized size of result for 412 is 301593
> 14/07/31 15:08:02 INFO Executor: Sending result for 412 directly to driver
> 14/07/31 15:08:02 INFO Executor: Finished task ID 412
> 14/07/31 15:08:04 INFO CoarseGrainedExecutorBackend: Got assigned task 437
> 14/07/31 15:08:04 INFO Executor: Running task ID 437
> 14/07/31 15:08:04 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 100 non-empty
blocks out of 100 blocks
> 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 16 remote
fetches in 6 ms
> 14/07/31 15:08:04 INFO Executor: Serialized size of result for 437 is 312543
> 14/07/31 15:08:04 INFO Executor: Sending result for 437 directly to driver
> 14/07/31 15:08:04 INFO Executor: Finished task ID 437
> 14/07/31 15:08:04 INFO CoarseGrainedExecutorBackend: Got assigned task 445
> 14/07/31 15:08:04 INFO Executor: Running task ID 445
> 14/07/31 15:08:04 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 100 non-empty
blocks out of 100 blocks
> 14/07/31 15:08:04 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 16 remote
fetches in 6 ms
> 14/07/31 15:08:04 INFO Executor: Serialized size of result for 445 is 307049
> 14/07/31 15:08:04 INFO Executor: Sending result for 445 directly to driver
> 14/07/31 15:08:04 INFO Executor: Finished task ID 445
> 14/07/31 15:08:06 INFO CoarseGrainedExecutorBackend: Got assigned task 467
> 14/07/31 15:08:06 INFO Executor: Running task ID 467
> 14/07/31 15:08:06 INFO BlockManager: Found block broadcast_0 locally
> 14/07/31 15:08:06 INFO BlockFetcherIterator$BasicBlockFetcherIterator: maxBytesInFlight:
50331648, targetRequestSize: 10066329
> 14/07/31 15:08:06 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 100 non-empty
blocks out of 100 blocks
> 14/07/31 15:08:06 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 16 remote
fetches in 6 ms
> 14/07/31 15:08:07 INFO Executor: Serialized size of result for 467 is 301177
> 14/07/31 15:08:07 INFO Executor: Sending result for 467 directly to driver
> 14/07/31 15:08:07 INFO Executor: Finished task ID 467
> 14/07/31 15:08:18 INFO ShuffleBlockManager: Deleted all files for shuffle 1
> 14/07/31 15:09:00 WARN BlockManagerMaster: Error sending message to BlockManagerMaster
in 1 attempts*java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
> 	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> 	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> 	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> 	at scala.concurrent.Await$.result(package.scala:107)
> 	at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:237)
> 	at org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:51)
> 	at org.apache.spark.storage.BlockManager.org <http://org.apache.spark.storage.BlockManager.org>$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:113)
> 	at org.apache.spark.storage.BlockManager$$anonfun$initialize$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(BlockManager.scala:158)
> 	at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:790)
> 	at org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:158)
> 	at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
> 	at akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)*
> 14/07/31 15:09:12 INFO ShuffleBlockManager: Deleted all files for shuffle 0
> 14/07/31 15:09:12 INFO BlockManager: Removing RDD 6
> 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_88
> 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_88 of size 1921571 dropped from memory
(free 12338538050)
> 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_25
> 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_25 of size 1922882 dropped from memory
(free 12340460932)
> 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_10
> 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_10 of size 1912396 dropped from memory
(free 12342373328)
> 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_54
> 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_54 of size 1909775 dropped from memory
(free 12344283103)
> 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_80
> 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_80 of size 1927469 dropped from memory
(free 12346210572)
> 14/07/31 15:09:12 INFO BlockManager: Removing block rdd_6_48
> 14/07/31 15:09:12 INFO MemoryStore: Block rdd_6_48 of size 1927469 dropped from memory
(free 12348138041)
>
>
>
>

Mime
View raw message