spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <mayur.rust...@gmail.com>
Subject Re: how to set task number?
Date Fri, 23 May 2014 08:37:52 GMT
How many cores do you see on your spark master (8080 port).
By default spark application should take all cores when you launch it.
Unless you have set max core configuration.


Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Thu, May 22, 2014 at 4:07 PM, qingyang li <liqingyang1985@gmail.com>wrote:

> my aim of setting task number is to increase the query speed,    and I
> have also found " mapPartitionsWithIndex at Operator.scala:333<http://192.168.1.101:4040/stages/stage?id=17>"
> is costing much time.  so, my another question is :
> how to tunning mapPartitionsWithIndex<http://192.168.1.101:4040/stages/stage?id=17>
> to make the costing time down?
>
>
>
>
> 2014-05-22 18:09 GMT+08:00 qingyang li <liqingyang1985@gmail.com>:
>
> i have added  SPARK_JAVA_OPTS+="-Dspark.
>> default.parallelism=40 "  in shark-env.sh,
>> but i find there are only10 tasks on the cluster and 2 tasks each machine.
>>
>>
>> 2014-05-22 18:07 GMT+08:00 qingyang li <liqingyang1985@gmail.com>:
>>
>> i have added  SPARK_JAVA_OPTS+="-Dspark.default.parallelism=40 "  in
>>> shark-env.sh
>>>
>>>
>>> 2014-05-22 17:50 GMT+08:00 qingyang li <liqingyang1985@gmail.com>:
>>>
>>> i am using tachyon as storage system and using to shark to query a table
>>>> which is a bigtable, i have 5 machines as a spark cluster, there are 4
>>>> cores on each machine .
>>>> My question is:
>>>> 1. how to set task number on each core?
>>>> 2. where to see how many partitions of one RDD?
>>>>
>>>
>>>
>>
>

Mime
View raw message