spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Xiao <xiaotao.cs....@gmail.com>
Subject Re: Reading from HBase is too slow
Date Wed, 08 Oct 2014 03:29:29 GMT
I found the reason why reading HBase is too slow.  Although each
regionserver serves multiple regions for the table I'm reading, the number
of Spark workers allocated by Yarn is too low. Actually, I could see that
the table has dozens of regions spread over about 20 regionservers, but
only two Spark workers are allocated by Yarn. What is worse, the two
workers run one after one. So, the Spark job lost parallelism.

*So now the question is : Why are only 2 workers allocated? *

The following is the log info in ApplicationMaster Log UI and we can see
that only 2 workers are allocated on two nodes (*a04.jsepc.com
<http://a04.jsepc.com>* and *b06 jsepc.com <http://jsepc.com>*)

Showing 4096 bytes. Click here for full log
erLauncher: ApplicationAttemptId: appattempt_1412731028648_0157_000001
14/10/08 09:55:16 INFO yarn.WorkerLauncher: Registering the
ApplicationMaster
14/10/08 09:55:16 INFO yarn.WorkerLauncher: Waiting for Spark driver to be
reachable.
14/10/08 09:55:16 INFO yarn.WorkerLauncher: Driver now available:
a04.jsepc.com:56888
14/10/08 09:55:16 INFO yarn.WorkerLauncher: Listen to driver: akka.tcp://
spark@a04.jsepc.com:56888/user/CoarseGrainedScheduler
14/10/08 09:55:16 INFO yarn.WorkerLauncher: *Allocating 2 workers*.
14/10/08 09:55:16 INFO yarn.YarnAllocationHandler: *Will Allocate 2 worker
containers, each with 1408 memory*
14/10/08 09:55:16 INFO yarn.YarnAllocationHandler: Container request (host:
Any, priority: 1, capability: <memory:1408, vCores:1>
14/10/08 09:55:16 INFO yarn.YarnAllocationHandler: Container request (host:
Any, priority: 1, capability: <memory:1408, vCores:1>
14/10/08 09:55:20 INFO util.RackResolver: *Resolved a04.jsepc.com
<http://a04.jsepc.com> to /rack1*
14/10/08 09:55:20 INFO util.RackResolver: *Resolved b06.jsepc.com
<http://b06.jsepc.com> to /rack2*
14/10/08 09:55:20 INFO yarn.YarnAllocationHandler: Launching container
container_1412731028648_0157_01_000002 for on host a04.jsepc.com
14/10/08 09:55:20 INFO yarn.YarnAllocationHandler: Launching
WorkerRunnable. driverUrl: akka.tcp://
spark@a04.jsepc.com:56888/user/CoarseGrainedScheduler,  workerHostname:
a04.jsepc.com
14/10/08 09:55:21 INFO yarn.YarnAllocationHandler: Launching container
container_1412731028648_0157_01_000003 for on host b06.jsepc.com
14/10/08 09:55:21 INFO yarn.YarnAllocationHandler: Launching
WorkerRunnable. driverUrl: akka.tcp://
spark@a04.jsepc.com:56888/user/CoarseGrainedScheduler,  workerHostname:
b06.jsepc.com
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Starting Worker Container
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Starting Worker Container
14/10/08 09:55:21 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-nodemanagers-proxies : 500
14/10/08 09:55:21 INFO impl.ContainerManagementProtocolProxy:
yarn.client.max-nodemanagers-proxies : 500
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Setting up
ContainerLaunchContext
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Setting up
ContainerLaunchContext
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Preparing Local resources
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Preparing Local resources
14/10/08 09:55:21 INFO yarn.WorkerLauncher: All workers have launched.
14/10/08 09:55:21 INFO yarn.WorkerLauncher: Started progress reporter
thread - sleep time : 5000
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Prepared Local resources
Map(spark.jar -> resource { scheme: "hdfs" host: "jsepc-ns" port: -1 file:
"/user/root/.sparkStaging/application_1412731028648_0157/spark-assembly_2.10-0.9.0-cdh5.0.1-hadoop2.3.0-cdh5.0.1.jar"
} size: 75288668 timestamp: 1412733307395 type: FILE visibility: PRIVATE)
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Prepared Local resources
Map(spark.jar -> resource { scheme: "hdfs" host: "jsepc-ns" port: -1 file:
"/user/root/.sparkStaging/application_1412731028648_0157/spark-assembly_2.10-0.9.0-cdh5.0.1-hadoop2.3.0-cdh5.0.1.jar"
} size: 75288668 timestamp: 1412733307395 type: FILE visibility: PRIVATE)
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Setting up worker with
commands: List($JAVA_HOME/bin/java -server  -XX:OnOutOfMemoryError='kill
%p' -Xms1024m -Xmx1024m  -Djava.io.tmpdir=$PWD/tmp
 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://
spark@a04.jsepc.com:56888/user/CoarseGrainedScheduler 2 b06.jsepc.com 1 1>
<LOG_DIR>/stdout 2> <LOG_DIR>/stderr)
14/10/08 09:55:21 INFO yarn.WorkerRunnable: Setting up worker with
commands: List($JAVA_HOME/bin/java -server  -XX:OnOutOfMemoryError='kill
%p' -Xms1024m -Xmx1024m  -Djava.io.tmpdir=$PWD/tmp
 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://
spark@a04.jsepc.com:56888/user/CoarseGrainedScheduler 1 a04.jsepc.com 1 1>
<LOG_DIR>/stdout 2> <LOG_DIR>/stderr)
14/10/08 09:55:21 INFO impl.ContainerManagementProtocolProxy: *Opening
proxy : a04.jsepc.com:8041 <http://a04.jsepc.com:8041>*
14/10/08 09:55:21 INFO impl.ContainerManagementProtocolProxy: *Opening
proxy : b06.jsepc.com:8041 <http://b06.jsepc.com:8041>*


 Here <http://pastebin.com/VhfmHPQe>is the log printed on console while the
Spark job is running.



2014-10-02 0:58 GMT+08:00 Vladimir Rodionov <vrodionov@splicemachine.com>:

> Yes, its in 0.98. CDH is free (w/o subscription) and sometimes its worth
> upgrading to the latest version (which is 0.98 based).
>
> -Vladimir Rodionov
>
> On Wed, Oct 1, 2014 at 9:52 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> As far as I know, that feature is not in CDH 5.0.0
>>
>> FYI
>>
>> On Wed, Oct 1, 2014 at 9:34 AM, Vladimir Rodionov <
>> vrodionov@splicemachine.com> wrote:
>>
>>> Using TableInputFormat is not the fastest way of reading data from
>>> HBase. Do not expect 100s of Mb per sec. You probably should take a look at
>>> M/R over HBase snapshots.
>>>
>>> https://issues.apache.org/jira/browse/HBASE-8369
>>>
>>> -Vladimir Rodionov
>>>
>>> On Wed, Oct 1, 2014 at 8:17 AM, Tao Xiao <xiaotao.cs.nju@gmail.com>
>>> wrote:
>>>
>>>> I can submit a MapReduce job reading that table, although its
>>>> processing rate is also a litter slower than I expected, but not that slow
>>>> as Spark.
>>>>
>>>>
>>>>
>>
>

Mime
View raw message