spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sven Krasser <kras...@gmail.com>
Subject Re: can I buffer flatMap input at each worker node?
Date Mon, 12 Jan 2015 18:22:03 GMT
Not sure I understand correctly, but it sounds like you're looking for
mapPartitions().
-Sven

On Mon, Jan 12, 2015 at 10:17 AM, maherrt <maherrt@hotmail.com> wrote:

> Dear All
>
> what i want to do is :
> as the data is partitioned on many worker nodes I want to be able to
> process
> this partition of data as a whole on each partition and then produce my
> output using flatMap for example.
> so can I loads all of the input records on one worker node and emitting any
> output using map function?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/can-I-buffer-flatMap-input-at-each-worker-node-tp21106.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
http://sites.google.com/site/krasser/?utm_source=sig

Mime
View raw message