spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sven Krasser <kras...@gmail.com>
Subject Re: Manually trigger RDD map function without action
Date Tue, 13 Jan 2015 01:57:00 GMT
Hey Kevin,

I assume you want to trigger the map() for a side effect (since you don't
care about the result). To Cody's point, you can use foreach() *instead* of
map(). So instead of e.g. x.map(a => foo(a)).foreach(a => a), you'd run
x.foreach(a => foo(a)).

Best,
-Sven

On Mon, Jan 12, 2015 at 5:13 PM, Kevin Jung <itsjb.jung@samsung.com> wrote:

> Cody said "If you don't care about the value that your map produced
> (because
> you're not already collecting or saving it), then is foreach more
> appropriate to what you're doing?" but I can not see it from this thread.
> Anyway, I performed small benchmark to test what function is the most
> efficient way. And a winner is foreach(a => a) according to everyone's
> expectations. Collect can cause OOM from driver and count is very slower
> than the others. Thanks all.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Manually-trigger-RDD-map-function-without-action-tp21094p21110.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
http://sites.google.com/site/krasser/?utm_source=sig

Mime
View raw message