spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Fregly <ch...@fregly.com>
Subject Re: multiple passes in mapPartitions
Date Wed, 02 Jul 2014 02:19:26 GMT
also, multiple calls to mapPartitions() will be pipelined by the spark
execution engine into a single stage, so the overhead is minimal.


On Fri, Jun 13, 2014 at 9:28 PM, zhen <z.he@latrobe.edu.au> wrote:

> Thank you for your suggestion. We will try it out and see how it performs.
> We
> think the single call to mapPartitions will be faster but we could be
> wrong.
> It would be nice to have a "clone method" on the iterator.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/multiple-passes-in-mapPartitions-tp7555p7616.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Mime
View raw message