spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <shushantaror...@gmail.com>
Subject user threads in executors
Date Tue, 21 Jul 2015 08:11:46 GMT
Hi

Can I create user threads in executors.
I have a streaming app where after processing I have a requirement to push
events to external system . Each post request costs ~90-100 ms.

To make post parllel, I can not use same thread because that is limited by
no of cores available in system , can I useuser therads in spark App? I
tried to create 2 thredas in a map tasks and it worked.

Is there any upper limit on no of user threds in spark executor ? Is it a
good idea to create user threads in spark map task?

Thanks

Mime
View raw message