spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <>
Subject Re: Replacing Spark's native scheduler with Sparrow
Date Sat, 08 Nov 2014 02:20:11 GMT
> If, for example, you have a cluster of 100 machines, this means the
> scheduler can launch 150 tasks per machine per second.

Did you mean 15 tasks per machine per second here? Or alternatively, 10

I don't know of any existing Spark clusters that have a large enough number
> of machines or short enough tasks to justify the added complexity of
> distributing the scheduler.

Actually, this was the reason I took interest in Sparrow--specifically, the
idea of a Spark cluster handling many very short (<< 50 ms) tasks.

At the recent Spark Committer Night
<> in NYC, I asked Michael
if he thought that Spark SQL could eventually completely fill the need for
very low latency queries currently served by MPP databases like Redshift or
Vertica. If I recall correctly, he said that the main obstacle to that was
simply task startup time, which is on the order of 100 ms.

Is there interest in (or perhaps an existing initiative related to)
improving task startup times to the point where one could legitimately look
at Spark SQL as a low latency database that can serve many users or
applications at once? That would probably make a good use case for Sparrow,


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message