spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Chen <tnac...@gmail.com>
Subject Re: Tuning Spark Streaming jobs
Date Mon, 22 Dec 2014 17:01:02 GMT
Hi Gerard,

Really nice guide!

I'm particularly interested in the Mesos scheduling side to more evenly distribute cores across
cluster.

I wonder if you are using coarse grain mode or fine grain mode? 

I'm making changes to the spark mesos scheduler and I think we can propose a best way to achieve
what you mentioned.

Tim

Sent from my iPhone

> On Dec 22, 2014, at 8:33 AM, Gerard Maas <gerard.maas@gmail.com> wrote:
> 
> Hi,
> 
> After facing issues with the performance of some of our Spark Streaming
> jobs, we invested quite some effort figuring out the factors that affect
> the performance characteristics of a Streaming job. We  defined an
> empirical model that helps us reason about Streaming jobs and applied it to
> tune the jobs in order to maximize throughput.
> 
> We have summarized our findings in a blog post with the intention of
> collecting feedback and hoping that it is useful to other Spark Streaming
> users facing similar issues.
> 
> http://www.virdata.com/tuning-spark/
> 
> Your feedback is welcome.
> 
> With kind regards,
> 
> Gerard.
> Data Processing Team Lead
> Virdata.com
> @maasg

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message