spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Goodson <martingood...@gmail.com>
Subject Re: Avoid broacasting huge variables
Date Sat, 20 Sep 2014 09:30:44 GMT
We normally copy a file to the nodes and then explicitly load it in a
function passed to mapPartitions.

On 9/20/14, octavian.ganea <octavian.ganea@inf.ethz.ch> wrote:
> Anyone ?
>
> Is there any option to load data in each node before starting any
> computation like it is the initialization of mappers in Hadoop ?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Avoid-broacasting-huge-variables-tp14696p14726.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
--
Martin Goodson
@martingoodson

   -

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message