spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vincent gromakowski <>
Subject Re: How to avoid unnecessary spark starkups on every request?
Date Wed, 02 Nov 2016 08:05:06 GMT
I am currently using akka http sending requests to multiple spark actors
that use a preloaded spark context and fair scheduler. It's only a
prototype and I haven't tested the concurrency but it seems one of the
rigth way to do. Complete processing time is arround 600 ms.The other way
would be to use a spark job server but i don't like to split my REST API in
2 (one business  in akka http and one technical in jobserver).

Le 2 nov. 2016 8:34 AM, "Fanjin Zeng" <> a écrit :

>  Hi,
>  I am working on a project that takes requests from HTTP server and
> computes accordingly on spark. But the problem is when I receive many
> request at the same time, users have to waste a lot of time on the
> unnecessary startups that occur on each request. Does Spark have built-in
> job scheduler function to solve this problem or is there any trick can be
> used to avoid these unnecessary startups?
>  Thanks,
>  Fanjin
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message