spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: number of executors
Date Mon, 18 May 2015 16:07:11 GMT
*All

On Mon, May 18, 2015 at 9:07 AM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:

> Hi Xiaohe,
>
> The all Spark options must go before the jar or they won't take effect.
>
> -Sandy
>
> On Sun, May 17, 2015 at 8:59 AM, xiaohe lan <zombiexcoder@gmail.com>
> wrote:
>
>> Sorry, them both are assigned task actually.
>>
>> Aggregated Metrics by Executor
>> Executor IDAddressTask TimeTotal TasksFailed TasksSucceeded TasksInput
>> Size / RecordsShuffle Write Size / RecordsShuffle Spill (Memory)Shuffle
>> Spill (Disk)1host1:61841.7 min505640.0 MB / 12318400382.3 MB / 121007701630.4
>> MB295.4 MB2host2:620721.7 min505640.0 MB / 12014510386.0 MB / 109269121646.6
>> MB304.8 MB
>>
>> On Sun, May 17, 2015 at 11:50 PM, xiaohe lan <zombiexcoder@gmail.com>
>> wrote:
>>
>>> bash-4.1$ ps aux | grep SparkSubmit
>>> xilan     1704 13.2  1.2 5275520 380244 pts/0  Sl+  08:39   0:13
>>> /scratch/xilan/jdk1.8.0_45/bin/java -cp
>>> /scratch/xilan/spark/conf:/scratch/xilan/spark/lib/spark-assembly-1.3.1-hadoop2.4.0.jar:/scratch/xilan/spark/lib/datanucleus-core-3.2.10.jar:/scratch/xilan/spark/lib/datanucleus-api-jdo-3.2.6.jar:/scratch/xilan/spark/lib/datanucleus-rdbms-3.2.9.jar:/scratch/xilan/hadoop/etc/hadoop
>>> -Xms512m -Xmx512m org.apache.spark.deploy.SparkSubmit --master yarn
>>> target/scala-2.10/simple-project_2.10-1.0.jar --class scala.SimpleApp
>>> --num-executors 5 --executor-cores 4
>>> xilan     1949  0.0  0.0 103292   800 pts/1    S+   08:40   0:00 grep
>>> --color SparkSubmit
>>>
>>>
>>> When look at the sparkui, I see the following:
>>> Aggregated Metrics by ExecutorExecutor IDAddressTask TimeTotal TasksFailed
>>> TasksSucceeded TasksShuffle Read Size / Records1host1:304836 s101127.1
>>> MB / 28089782host2:49970 ms00063.4 MB / 1810945
>>>
>>> So executor 2 is not even assigned a task ? Maybe I have some problems
>>> in my setting, but I don't know what could be the possible settings I set
>>> wrong or have not set.
>>>
>>>
>>> Thanks,
>>> Xiaohe
>>>
>>> On Sun, May 17, 2015 at 11:16 PM, Akhil Das <akhil@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> Did you try --executor-cores param? While you submit the job, do a ps
>>>> aux | grep spark-submit and see the exact command parameters.
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Sat, May 16, 2015 at 12:31 PM, xiaohe lan <zombiexcoder@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a 5 nodes yarn cluster, I used spark-submit to submit a simple
>>>>> app.
>>>>>
>>>>>  spark-submit --master yarn
>>>>> target/scala-2.10/simple-project_2.10-1.0.jar --class scala.SimpleApp
>>>>> --num-executors 5
>>>>>
>>>>> I have set the number of executor to 5, but from sparkui I could see
>>>>> only two executors and it ran very slow. What did I miss ?
>>>>>
>>>>> Thanks,
>>>>> Xiaohe
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message