spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Marscher <rmarsc...@localytics.com>
Subject Re: user threads in executors
Date Tue, 21 Jul 2015 16:35:07 GMT
You can certainly create threads in a map transformation. We do this to do
concurrent DB lookups during one stage for example. I would recommend,
however, that you switch to mapPartitions from map as this allows you to
create a fixed size thread pool to share across items on a partition as
opposed to spawning a future per record in the RDD for example.

On Tue, Jul 21, 2015 at 4:11 AM, Shushant Arora <shushantarora09@gmail.com>
wrote:

> Hi
>
> Can I create user threads in executors.
> I have a streaming app where after processing I have a requirement to push
> events to external system . Each post request costs ~90-100 ms.
>
> To make post parllel, I can not use same thread because that is limited by
> no of cores available in system , can I useuser therads in spark App? I
> tried to create 2 thredas in a map tasks and it worked.
>
> Is there any upper limit on no of user threds in spark executor ? Is it a
> good idea to create user threads in spark map task?
>
> Thanks
>
>


-- 
*Richard Marscher*
Software Engineer
Localytics
Localytics.com <http://localytics.com/> | Our Blog
<http://localytics.com/blog> | Twitter <http://twitter.com/localytics> |
Facebook <http://facebook.com/localytics> | LinkedIn
<http://www.linkedin.com/company/1148792?trk=tyah>

Mime
View raw message