spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soren Macbeth <so...@yieldbot.com>
Subject Requirements of objects stored in RDDs
Date Thu, 08 May 2014 16:55:05 GMT
Hi,

What are the requirements of objects that are stored in RDDs?

I'm still struggling with an exception I've already posted about several
times. My questions are:

1) What interfaces are objects stored in RDDs expected to implement, if any?
2) Are collections (be they scala, java or otherwise) handled differently
than other objects?

The bug I'm hitting is when I try to use my clojure DSL (which wraps the
java api) with clojure collections, specifically
clojure.lang.PersistentVectors in my RDDs. Here is the exception message:

org.apache.spark.SparkException: Job aborted: Exception while deserializing
and fetching task: com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Can not set final scala.collecti
on.convert.Wrappers field
scala.collection.convert.Wrappers$SeqWrapper.$outer to
clojure.lang.PersistentVector

Now, this same application works fine in local mode and tests, but it fails
when run under mesos. That would seem to me to point to something around
RDD partitioning for tasks, but I'm not sure.

I don't know much scala, but according to google, SeqWrapper is part of the
implicit JavaConversion functionality of scala collections. Under what
circumstances would spark be trying to wrap my RDD objects in scala
collections?

Finally - I'd like to point out that this is not a serialization issue with
my clojure collection objects. I have registered serializers for them and
have verified they serialize and deserialize perfectly well in spark.

One last note is that this failure occurs after all the tasks for finished
for a reduce stage and the results are returned to the driver.

TIA

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message