spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <r...@databricks.com>
Subject Re: bug using kryo as closure serializer
Date Mon, 05 May 2014 05:24:22 GMT
Technically you only need to change the build file, and change part of a
line in SparkEnv so you don't have to break your oath :)



On Sun, May 4, 2014 at 10:22 PM, Soren Macbeth <soren@yieldbot.com> wrote:

> that would violate my personal oath of never writing a single line of
> scala, but I might be able to do that if I can get past the issue this
> issue I'm struggling with in this thread.
>
> On Sunday, May 4, 2014, Reynold Xin <rxin@databricks.com> wrote:
>
> > Thanks. Do you mind playing with chill-scala a little bit and see if it
> > actually works well for closures? One way to try is to hard code the
> > serializer to use Kryo with chill-scala, and then run through all the
> unit
> > tests.
> >
> > If it works well, we can incorporate that in the next release (probably
> not
> > 1.0, but after that).
> >
> >
> > On Sun, May 4, 2014 at 9:08 PM, Soren Macbeth <soren@yieldbot.com>
> wrote:
> >
> > > fwiw, it seems like it wouldn't be very difficult to integrate
> > chill-scala,
> > > since you're already chill-java and probably get kryo serialization of
> > > closures and all sorts of other scala stuff for free. All that would be
> > > needed would be to include the dependency and then update
> KryoSerializer
> > to
> > > register the stuff in chill-scala.
> > >
> > > In that case, you could probably safely make kryo the default
> serializer,
> > > which I think would be desirable in general.
> > >
> > >
> > > On Sun, May 4, 2014 at 8:48 PM, Reynold Xin <rxin@databricks.com>
> wrote:
> > >
> > > > Good idea. I submitted a pull request for the doc update here:
> > > > https://github.com/apache/spark/pull/642
> > > >
> > > >
> > > > On Sun, May 4, 2014 at 3:54 PM, Soren Macbeth <soren@yieldbot.com>
> > > wrote:
> > > >
> > > > > Thanks for the reply!
> > > > >
> > > > > Ok, if that's the case, I'd recommend a note to that affect in the
> > docs
> > > > at
> > > > > least.
> > > > >
> > > > > Just to give some more context here, I'm working on a Clojure DSL
> for
> > > > Spark
> > > > > called Flambo, which I plan to open source shortly. If I could I'd
> > like
> > > > to
> > > > > focus on the initial bug that I hit.
> > > > >
> > > > > Exception in thread "main" org.apache.spark.SparkException: Job
> > > aborted:
> > > > > Exception while deserializing and fetching task:
> > > > > com.esotericsoftware.kryo.KryoException:
> > > > > java.lang.IllegalArgumentException: Can not set final
> > > > > scala.collection.convert.Wrappers field
> > > > > scala.collection.convert.Wrappers$SeqWrapper.$outer to
> > > > > clojure.lang.PersistentVector
> > > > > Serialization trace:
> > > > > $outer (scala.collection.convert.Wrappers$SeqWrapper)
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> >
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> > > > >         at
> > > > > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> > > > >         at org.apache.spark.scheduler.DAGScheduler.org
> > > > >
> > >
> $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
> > > > >         at scala.Option.foreach(Option.scala:236)
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
> > > > >         at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
> > > > >         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> > > > >         at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> > > > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message