Do you mind sharing what your software does? What is the input data size? What is the spark version and apis used? How many nodes? What is the input data format? Is compression used?

On 21 Sep 2016, at 13:37, Trinadh Kaja <> wrote:

Hi all,

how to increase spark performance ,i am using pyspark.

cluster info :

Total memory :600gb
Cores            :96

command :
spark-submit --master  yarn-client --executor-memory 10G --num-executors 50 --executor-cores 2 --driver-memory 10g --queue thequeue

please help on this