spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Spark app performance
Date Thu, 01 Jan 2015 08:35:09 GMT
Would be great if you can share the piece of code happening inside your
mapPartition, I'm assuming you are creating/handling a lot of Complex
objects and hence it slows down the performance. Here's a link
<http://spark.apache.org/docs/latest/tuning.html> to performance tuning if
you haven't seen it already.

Thanks
Best Regards

On Wed, Dec 31, 2014 at 8:45 AM, Raghavendra Pandey <
raghavendra.pandey@gmail.com> wrote:

> I have a spark app that involves series of mapPartition operations and
> then a keyBy operation. I have measured the time inside mapPartition
> function block. These blocks take trivial time. Still the application takes
> way too much time and even sparkUI shows that much time.
> So i was wondering where does it take time and how can I reduce this.
>
> Thanks
> Raghavendra
>

Mime
View raw message