spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sai Prasanna <ansaiprasa...@gmail.com>
Subject Spark executor memory & relationship with worker threads
Date Wed, 26 Mar 2014 05:36:38 GMT
Hi All,
Does number of worker threads bear any relationship to setting executor
memory ?.
I have a 16 GB RAM, with an 8-core processor. I had set SPARK_MEM to 12g
and was running locally with default 1 thread.
So this means there can be maximum one executor in one node scheduled at
any point of time. If I increase the number of worker threads to say 4, do
I need to reduce SPARK_MEM to 3g or I need not ???

Is there any performance difference between running in interactive spark
shell to non-interactive standalone spark apart from building time(sbt…)
???

THANKS !!

Mime
View raw message