spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Issue with Spark latest 1.2.0 build - ClassCastException from [B to SerializableWritable
Date Wed, 26 Nov 2014 12:03:22 GMT
I'll take a wild guess that you have mismatching versions of Spark at
play. Your cluster has one build and you're accidentally including
another version.

I think this code path has changed recently (
https://github.com/apache/spark/commit/7e63bb49c526c3f872619ae14e4b5273f4c535e9#diff-83eb37f7b0ebed3c14ccb7bff0d577c2
?) but shouldn't cause this kind of problem per se, I think. Maybe
double-check the version mismatch possibility first.

On Wed, Nov 26, 2014 at 2:42 AM, lokeshkumar <lokesh@dataken.net> wrote:
> Hello forum,
>
> We are using spark distro built from the source of latest 1.2.0 tag.
> And we are facing the below issue, while trying to act upon the JavaRDD
> instance, the stacktrace is given below.
> Can anyone please let me know, what can be wrong here?
>
> java.lang.ClassCastException: [B cannot be cast to
> org.apache.spark.SerializableWritable
>         at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:138)
>         at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
>         at scala.Option.getOrElse(Option.scala:120)
>         at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
>         at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
>         at scala.Option.getOrElse(Option.scala:120)
>         at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
>         at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
>         at scala.Option.getOrElse(Option.scala:120)
>         at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
>         at org.apache.spark.rdd.RDD.take(RDD.scala:1060)
>         at org.apache.spark.api.java.JavaRDDLike$class.take(JavaRDDLike.scala:419)
>         at org.apache.spark.api.java.JavaRDD.take(JavaRDD.scala:32)
>         at
> com.dataken.common.chores.InformationDataLoadChore.run(InformationDataLoadChore.java:69)
>         at com.dataken.common.pipeline.DatakenTask.start(DatakenTask.java:110)
>         at
> com.dataken.tasks.objectcentricprocessor.ObjectCentricProcessTask.execute(ObjectCentricProcessTask.java:99)
>         at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>         at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2014-11-26 08:07:38,454 ERROR [DefaultQuartzScheduler_Worker-2]
> org.quartz.core.ErrorLogger
> Job (report_report.report_report threw an exception.
>
> org.quartz.SchedulerException: Job threw an unhandled exception. [See nested
> exception: java.lang.ClassCastException: [B cannot be cast to
> org.apache.spark.SerializableWritable]
>         at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
>         at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.ClassCastException: [B cannot be cast to
> org.apache.spark.SerializableWritable
>         at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:138)
>         at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
>         at scala.Option.getOrElse(Option.scala:120)
>         at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
>         at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
>         at scala.Option.getOrElse(Option.scala:120)
>         at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
>         at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
>         at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
>         at scala.Option.getOrElse(Option.scala:120)
>         at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
>         at org.apache.spark.rdd.RDD.take(RDD.scala:1060)
>         at org.apache.spark.api.java.JavaRDDLike$class.take(JavaRDDLike.scala:419)
>         at org.apache.spark.api.java.JavaRDD.take(JavaRDD.scala:32)
>         at
> com.dataken.common.chores.InformationDataLoadChore.run(InformationDataLoadChore.java:69)
>         at com.dataken.common.pipeline.DatakenTask.start(DatakenTask.java:110)
>         at
> com.dataken.tasks.objectcentricprocessor.ObjectCentricProcessTask.execute(ObjectCentricProcessTask.java:99)
>         at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>         ... 1 more
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issue-with-Spark-latest-1-2-0-build-ClassCastException-from-B-to-SerializableWritable-tp19815.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message