spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xiaohe lan <zombiexco...@gmail.com>
Subject Re: number of executors
Date Sun, 17 May 2015 15:59:06 GMT
Sorry, them both are assigned task actually.

Aggregated Metrics by Executor
Executor IDAddressTask TimeTotal TasksFailed TasksSucceeded TasksInput Size
/ RecordsShuffle Write Size / RecordsShuffle Spill (Memory)Shuffle Spill
(Disk)1host1:61841.7 min505640.0 MB / 12318400382.3 MB / 121007701630.4 MB295.4
MB2host2:620721.7 min505640.0 MB / 12014510386.0 MB / 109269121646.6 MB304.8
MB

On Sun, May 17, 2015 at 11:50 PM, xiaohe lan <zombiexcoder@gmail.com> wrote:

> bash-4.1$ ps aux | grep SparkSubmit
> xilan     1704 13.2  1.2 5275520 380244 pts/0  Sl+  08:39   0:13
> /scratch/xilan/jdk1.8.0_45/bin/java -cp
> /scratch/xilan/spark/conf:/scratch/xilan/spark/lib/spark-assembly-1.3.1-hadoop2.4.0.jar:/scratch/xilan/spark/lib/datanucleus-core-3.2.10.jar:/scratch/xilan/spark/lib/datanucleus-api-jdo-3.2.6.jar:/scratch/xilan/spark/lib/datanucleus-rdbms-3.2.9.jar:/scratch/xilan/hadoop/etc/hadoop
> -Xms512m -Xmx512m org.apache.spark.deploy.SparkSubmit --master yarn
> target/scala-2.10/simple-project_2.10-1.0.jar --class scala.SimpleApp
> --num-executors 5 --executor-cores 4
> xilan     1949  0.0  0.0 103292   800 pts/1    S+   08:40   0:00 grep
> --color SparkSubmit
>
>
> When look at the sparkui, I see the following:
> Aggregated Metrics by ExecutorExecutor IDAddressTask TimeTotal TasksFailed
> TasksSucceeded TasksShuffle Read Size / Records1host1:304836 s101127.1 MB
> / 28089782host2:49970 ms00063.4 MB / 1810945
>
> So executor 2 is not even assigned a task ? Maybe I have some problems in
> my setting, but I don't know what could be the possible settings I set
> wrong or have not set.
>
>
> Thanks,
> Xiaohe
>
> On Sun, May 17, 2015 at 11:16 PM, Akhil Das <akhil@sigmoidanalytics.com>
> wrote:
>
>> Did you try --executor-cores param? While you submit the job, do a ps aux
>> | grep spark-submit and see the exact command parameters.
>>
>> Thanks
>> Best Regards
>>
>> On Sat, May 16, 2015 at 12:31 PM, xiaohe lan <zombiexcoder@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a 5 nodes yarn cluster, I used spark-submit to submit a simple
>>> app.
>>>
>>>  spark-submit --master yarn
>>> target/scala-2.10/simple-project_2.10-1.0.jar --class scala.SimpleApp
>>> --num-executors 5
>>>
>>> I have set the number of executor to 5, but from sparkui I could see
>>> only two executors and it ran very slow. What did I miss ?
>>>
>>> Thanks,
>>> Xiaohe
>>>
>>
>>
>

Mime
View raw message