spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
Subject How does executor cores change the spark job behavior ?
Date Mon, 06 Jul 2015 22:36:30 GMT
I have a simple job , that reads data => union => filter => map and the
count

1 Job started 2402 tasks read 149G of input.

I started the job with different number of executors

1) 1 -->  8.3 mins
2) 2 --> 5.6 mins
3) 3 --> 3.1 mins

1) Why is increasing the cores speading up this app ?
2) I started the job with --num-executors 9973 but when i click executors
tab i see 330 executors. So can i start the job with --num-executors 330 as
i get only that from YARN cluster ?
3) I had set the split size to 64 MB but when i start the job
with --executor-memory 14g , how do i decide how much memory i need ? also
as the cores are increasing how do i get that into the calculations ?
4) as the speed is getting better how far can i go with increasing
executors ?

-- 
Deepak

Mime
View raw message