spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Create/shutdown objects before/after RDD use (or: Non-serializable classes)
Date Mon, 02 Jun 2014 01:13:22 GMT
Xiangrui,

thanks for your suggestion!

On Sat, May 31, 2014 at 6:12 PM, Xiangrui Meng <mengxr@gmail.com> wrote:
> One hack you can try is:
>
> rdd.mapPartitions(iter => {
>   val x = new X()
>   iter.map(row => x.doSomethingWith(row)) ++ { x.shutdown(); Iterator.empty }
> })

In fact, I employed a similar hack by now:

rdd.mapPartitions(iter => {
  val x = new X()
  iter.map(row => {
    x.doSomethingWith(row)
    if (!iter.hasNext) x.shutdown()
    row
  })
})

Thanks
Tobias

Mime
View raw message