spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Chen <>
Subject Re: Tuning Spark Streaming jobs
Date Mon, 22 Dec 2014 17:01:02 GMT
Hi Gerard,

Really nice guide!

I'm particularly interested in the Mesos scheduling side to more evenly distribute cores across

I wonder if you are using coarse grain mode or fine grain mode? 

I'm making changes to the spark mesos scheduler and I think we can propose a best way to achieve
what you mentioned.


Sent from my iPhone

> On Dec 22, 2014, at 8:33 AM, Gerard Maas <> wrote:
> Hi,
> After facing issues with the performance of some of our Spark Streaming
> jobs, we invested quite some effort figuring out the factors that affect
> the performance characteristics of a Streaming job. We  defined an
> empirical model that helps us reason about Streaming jobs and applied it to
> tune the jobs in order to maximize throughput.
> We have summarized our findings in a blog post with the intention of
> collecting feedback and hoping that it is useful to other Spark Streaming
> users facing similar issues.
> Your feedback is welcome.
> With kind regards,
> Gerard.
> Data Processing Team Lead
> @maasg

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message