spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From map reduced <k3t.gi...@gmail.com>
Subject Re: Increasing Executor threadpool
Date Thu, 03 Nov 2016 07:09:31 GMT
The reason I am asking this is because I am sending (a fraction of) these
messages processed to a HTTP endpoint which has avg latency of around
70-90ms - and I have blocking calls as of now - which only with 7-10
threads (for 7- 10 cores) is slowing down everything. What would you
suggest? Going async?

On Wed, Nov 2, 2016 at 9:48 PM, map reduced <k3t.git.1@gmail.com> wrote:

> Hi,
>
> I am noticing that when there are N cores per executor, each executor only
> starts N threads to process the data (so 1 thread per core). Is there a way
> to increase more than N threads, i.e. say N+m threads per core?
>
> So I assigned 7 cores/executor, so I see 7 Active Tasks at all times.
>
> [image: Inline image 1]
>
> And only 7 threads doing all the work:
>
> [image: Inline image 2]
>
> Is there any way to make it atleast 2 threads/core?
>
> P.S.: Long running Streaming job, Standalone 2.0.0 cluster
>
> Thanks,
> KP
>

Mime
View raw message