spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <>
Subject Re: Spark app performance
Date Thu, 01 Jan 2015 08:35:09 GMT
Would be great if you can share the piece of code happening inside your
mapPartition, I'm assuming you are creating/handling a lot of Complex
objects and hence it slows down the performance. Here's a link
<> to performance tuning if
you haven't seen it already.

Best Regards

On Wed, Dec 31, 2014 at 8:45 AM, Raghavendra Pandey <> wrote:

> I have a spark app that involves series of mapPartition operations and
> then a keyBy operation. I have measured the time inside mapPartition
> function block. These blocks take trivial time. Still the application takes
> way too much time and even sparkUI shows that much time.
> So i was wondering where does it take time and how can I reduce this.
> Thanks
> Raghavendra

View raw message