spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Does Spark optimization might miss to run transformation?
Date Wed, 12 Aug 2015 18:29:13 GMT
-dev

If you want to guarantee the side effects happen you should use foreach or
foreachPartitions.  A `take`, for example, might only evaluate a subset of
the partitions until it find enough results.

On Wed, Aug 12, 2015 at 7:06 AM, Eugene Morozov <fathersson@list.ru> wrote:

> Hi!
>
> I’d like to complete action (store / print smth) inside of transformation
> (map or mapPartitions). This approach has some flaws, but there is a
> question. Might it happen that Spark will optimise (RDD or DataFrame)
> processing so that my mapPartitions simply won’t happen?
>
> --
> Eugene Morozov
> fathersson@list.ru
>
>
>
>
>

Mime
View raw message