Right, I understand that. I was just hoping to increase # of threads in this threadpool https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L85 somehow.

On Thu, Nov 3, 2016 at 12:55 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
The core is equivalent to the number of logical processeors

cat /proc/cpuinfo|grep processor|wc -l

That will tell you how many logical processors you have.

I gave an explanation for this a while back. As you are running in Standalone mode, this is my take:

Standalone mode

Resources are managed by Spark resource manager itself. You start your master and slaves/worker processes As far as I have worked it out the following applies

num-executors         --> It does not care about this. The number of executors will be the number of workers on each node
executor-memory       --> If you have set up SPARK_WORKER_MEMORY in spark-env.sh, this will be the memory used by the executor
executor-cores        --> If you have set up SPARK_WORKER_CORES in spark-env.sh, this will be the number of cores used by each executor
SPARK_WORKER_CORES=n ##, total number of cores to be used by executors by each worker
SPARK_WORKER_MEMORY=mg ##, to set how much total memory workers have to give executors (e.g. 1000m, 2g)


Dr Mich Talebzadeh


LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.


On 3 November 2016 at 04:48, map reduced <k3t.git.1@gmail.com> wrote:

I am noticing that when there are N cores per executor, each executor only starts N threads to process the data (so 1 thread per core). Is there a way to increase more than N threads, i.e. say N+m threads per core?

So I assigned 7 cores/executor, so I see 7 Active Tasks at all times.

Inline image 1

And only 7 threads doing all the work:

Inline image 2

Is there any way to make it atleast 2 threads/core?

P.S.: Long running Streaming job, Standalone 2.0.0 cluster