It might be due to the object is nested within some class which may not be serializable.

Also you can run the appluication using this jvm parameter to see detailed info about serialization -Dsun.io.serialization.extendedDebugInfo=true

On Fri, Oct 17, 2014 at 4:07 PM, shahab <shahab.mokari@gmail.com> wrote:
Hi,

Probably I am missing very simple principle , but something is wrong with my filter,
i get "org.apache.spark.SparkException: Task not serializable" expetion.

here is my filter function:
object OBJ { 
   def f1(): Boolean = { 
         var i = 1; 
         for (j<-1 to 10) i = i +1; 
         true;
   } 
}

rdd.filter(row => OBJ.f1())


And when I run, I get the following exception:

org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
at org.apache.spark.SparkContext.clean(SparkContext.scala:1242)
at org.apache.spark.rdd.RDD.filter(RDD.scala:282)
.......
Caused by: java.io.NotSerializableException: org.apache.spark.SparkConf
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
...........



best,
/Shahab




--

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·

sourav.chandra@livestream.com

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com