spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Increasing Executor threadpool
Date Thu, 03 Nov 2016 07:55:57 GMT
The core is equivalent to the number of logical processeors

cat /proc/cpuinfo|grep processor|wc -l


That will tell you how many logical processors you have.

I gave an explanation for this a while back. As you are running in
Standalone mode, this is my take:

Standalone mode
Resources are managed by Spark resource manager itself. You start your
master and slaves/worker processes As far as I have worked it out the
following applies

num-executors         --> It does not care about this. The number of
executors will be the number of workers on each node
executor-memory       --> If you have set up SPARK_WORKER_MEMORY in
spark-env.sh, this will be the memory used by the executor
executor-cores        --> If you have set up SPARK_WORKER_CORES in
spark-env.sh, this will be the number of cores used by each executor
SPARK_WORKER_CORES=n ##, total number of cores to be used by executors by
each worker
SPARK_WORKER_MEMORY=mg ##, to set how much total memory workers have to
give executors (e.g. 1000m, 2g)


HTH






Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 3 November 2016 at 04:48, map reduced <k3t.git.1@gmail.com> wrote:

> Hi,
>
> I am noticing that when there are N cores per executor, each executor only
> starts N threads to process the data (so 1 thread per core). Is there a way
> to increase more than N threads, i.e. say N+m threads per core?
>
> So I assigned 7 cores/executor, so I see 7 Active Tasks at all times.
>
> [image: Inline image 1]
>
> And only 7 threads doing all the work:
>
> [image: Inline image 2]
>
> Is there any way to make it atleast 2 threads/core?
>
> P.S.: Long running Streaming job, Standalone 2.0.0 cluster
>
> Thanks,
> KP
>

Mime
View raw message