spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourav Chandra <sourav.chan...@livestream.com>
Subject Re: What's wrong with my spark filter? I get "org.apache.spark.SparkException: Task not serializable"
Date Fri, 17 Oct 2014 10:41:08 GMT
It might be due to the object is nested within some class which may not be
serializable.

Also you can run the appluication using this jvm parameter to see detailed
info about serialization -Dsun.io.serialization.extendedDebugInfo=true

On Fri, Oct 17, 2014 at 4:07 PM, shahab <shahab.mokari@gmail.com> wrote:

> Hi,
>
> Probably I am missing very simple principle , but something is wrong with
> my filter,
> i get "org.apache.spark.SparkException: Task not serializable" expetion.
>
> here is my filter function:
> object OBJ {
>    def f1(): Boolean = {
>          var i = 1;
>          for (j<-1 to 10) i = i +1;
>          true;
>    }
> }
>
> rdd.filter(row => OBJ.f1())
>
>
> And when I run, I get the following exception:
>
> org.apache.spark.SparkException: Task not serializable
> at
> org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
> at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
> at org.apache.spark.SparkContext.clean(SparkContext.scala:1242)
> at org.apache.spark.rdd.RDD.filter(RDD.scala:282)
> .......
> Caused by: java.io.NotSerializableException: org.apache.spark.SparkConf
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> ...........
>
>
>
> best,
> /Shahab
>
>


-- 

Sourav Chandra

Senior Software Engineer

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
· · ·

sourav.chandra@livestream.com

o: +91 80 4121 8723

m: +91 988 699 3746

skype: sourav.chandra

Livestream

"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,

Bangalore 560034

www.livestream.com

Mime
View raw message