spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Marscher <>
Subject Re: user threads in executors
Date Tue, 21 Jul 2015 16:35:07 GMT
You can certainly create threads in a map transformation. We do this to do
concurrent DB lookups during one stage for example. I would recommend,
however, that you switch to mapPartitions from map as this allows you to
create a fixed size thread pool to share across items on a partition as
opposed to spawning a future per record in the RDD for example.

On Tue, Jul 21, 2015 at 4:11 AM, Shushant Arora <>

> Hi
> Can I create user threads in executors.
> I have a streaming app where after processing I have a requirement to push
> events to external system . Each post request costs ~90-100 ms.
> To make post parllel, I can not use same thread because that is limited by
> no of cores available in system , can I useuser therads in spark App? I
> tried to create 2 thredas in a map tasks and it worked.
> Is there any upper limit on no of user threds in spark executor ? Is it a
> good idea to create user threads in spark map task?
> Thanks

*Richard Marscher*
Software Engineer
Localytics <> | Our Blog
<> | Twitter <> |
Facebook <> | LinkedIn

View raw message