spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Replacing Spark's native scheduler with Sparrow
Date Mon, 10 Nov 2014 20:58:57 GMT
Too bad Nick, I dont have anything immediately ready that tests Spark
Streaming with those extreme settings. :)

On Mon, Nov 10, 2014 at 9:56 AM, Nicholas Chammas
<nicholas.chammas@gmail.com> wrote:
> On Sun, Nov 9, 2014 at 1:51 AM, Tathagata Das <tathagata.das1565@gmail.com>
> wrote:
>>
>> This causes a scalability vs. latency tradeoff - if your limit is 1000
>> tasks per second (simplifying from 1500), you could either configure
>> it to use 100 receivers at 100 ms batches (10 blocks/sec), or 1000
>> receivers at 1 second batches.
>
>
> This raises an interesting question, TD.
>
> Do we have a benchmark for Spark Streaming that tests it at the extreme for
> some key metric, perhaps processed messages per second per node? Something
> that would press Spark's ability to process tasks quickly enough.
>
> Given such a benchmark, it would probably be interesting to see how -- if at
> all -- Sparrow has an impact on Spark Streaming's performance.
>
> Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message