spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ray Melton <rtmel...@gmail.com>
Subject Re: init / shutdown for complex map job?
Date Sun, 28 Dec 2014 14:41:21 GMT
A follow-up to the blog cited below was hinted at, per "But Wait,
There's More ... To keep this post brief, the remainder will be left to
a follow-up post."

Is this follow-up pending?  Is it sort of pending?  Did the follow-up
happen, but I just couldn't find it on the web?

Regards, Ray.


On Sun, 28 Dec 2014 08:54:13 +0000
Sean Owen <sowen@cloudera.com> wrote:

> You can't quite do cleanup in mapPartitions in that way. Here is a
> bit more explanation (farther down):
> http://blog.cloudera.com/blog/2014/09/how-to-translate-from-mapreduce-to-apache-spark/
> On Dec 28, 2014 8:18 AM, "Akhil Das" <akhil@sigmoidanalytics.com>
> wrote:
> 
> > Something like?
> >
> > val a = myRDD.mapPartitions(p => {
> >
> >
> >
> >             //Do the init
> >
> >             //Perform some operations
> >
> >             //Shut it down?
> >
> >          })
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message