spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: [SPARK on MESOS] Avoid re-fetching Spark binary
Date Sun, 08 Jul 2018 05:31:23 GMT
Essentially correct. The latency to start a Spark Job is nowhere close to
2-4 seconds under typical conditions. Creating a new Spark Application
every time instead of running multiple Jobs in one Application is not going
to lead to acceptable interactive or real-time performance, nor is that an
execution model that Spark is ever likely to support in trying to meet
low-latency requirements. As such, reducing Application startup time (not
Job startup time) is not a priority.

On Fri, Jul 6, 2018 at 4:06 PM Timothy Chen <tnachen@gmail.com> wrote:

> I know there are some community efforts shown in Spark summits before,
> mostly around reusing the same Spark context with multiple “jobs”.
>
> I don’t think reducing Spark job startup time is a community priority
> afaik.
>
> Tim
> On Fri, Jul 6, 2018 at 7:12 PM Tien Dat <tphan.dat@gmail.com> wrote:
>
>> Dear Timothy,
>>
>> It works like a charm now.
>>
>> BTW (don't judge me if I am to greedy :-)), the latency to start a Spark
>> job
>> is around 2-4 seconds, unless I am not aware of some awesome optimization
>> on
>> Spark. Do you know if Spark community is working on reducing this latency?
>>
>> Best
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>

Mime
View raw message