spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Domen Grabec <do...@celtra.com>
Subject closure scope & Serialization
Date Wed, 19 Mar 2014 17:04:53 GMT
Hey,

I have 3 classes where 2 are in circular dependency like this:

package org.example
import org.apache.spark.SparkContext

class A(bLazy: => Option[B]) extends java.io.Serializable{
  lazy val b: Option[B] = bLazy
}

class B(aLazy: => Option[A]) extends java.io.Serializable{
  lazy val a: Option[A] = aLazy
}

class DoNotSerialize{

  def createAnObject(): A = {
    lazy val a: A = new A(Some(b)) // Requires class DoNotSerialize to
implement java.io.Serializable
    //lazy val a: A = new A(None) // Works OK
    lazy val b: B = new B(Some(a))
    a
  }
}

object main extends App{
 val dont = new DoNotSerialize()
 val sc = new SparkContext("local[8]", "test session")
 val scRdd = sc.parallelize(Seq(dont.createAnObject()))
 scRdd.count()
}

can someone please help me understand why in the first case DoNotSerialize
class needs to implement Serializable?

Regards, Domen

Mime
View raw message