spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: How to trace/debug serialization?
Date Thu, 06 Nov 2014 06:54:49 GMT
If you run the JVM with -Dsun.io.serialization.extendedDebugInfo=true
it will dump the 'path' to the unserializable object when an error
occurs.

On Wed, Nov 5, 2014 at 11:56 PM, ankits <ankitsoni9@gmail.com> wrote:
> In my spark job, I have a loop something like this:
>
> bla.forEachRdd(rdd => {
>   //init some vars
>   rdd.forEachPartition(partiton => {
>     //init some vars
>     partition.foreach(kv => {
>      ...
>
> I am seeing serialization errors (unread block data), because I think spark
> is trying to serialize the whole containing class. But I have been careful
> not to reference instance vars in the block.
>
> Is there a way to see exactly what class is failing serialization, and maybe
> how spark decided it needs to be serialized?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-trace-debug-serialization-tp18230.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message