spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soren Macbeth <so...@yieldbot.com>
Subject Re: ClassTag in Serializer in 1.0.0 makes non-scala callers sad panda
Date Mon, 02 Jun 2014 00:42:16 GMT
Yep, that's what I'm doing.

(def OBJECT-CLASS-TAG (.apply ClassTag$/MODULE$ java.lang.Object))

ps - I'm planning to open source this Clojure DSL soon as well


On Sun, Jun 1, 2014 at 5:10 PM, Matei Zaharia <matei.zaharia@gmail.com>
wrote:

> Ah, got it. In general it will always be safe to pass the ClassTag for
> java.lang.Object here — this is what our Java API does to say that type
> info is not known. So you can always pass that. Look at the Java code for
> how to get this ClassTag.
>
> Matei
>
> On Jun 1, 2014, at 4:33 PM, Soren Macbeth <soren@yieldbot.com> wrote:
>
> > I'm writing a Clojure DSL for Spark. I use kryo to serialize my clojure
> > functions and for efficiency I hook into Spark's kryo serializer. In
> order
> > to do that I get a SerializerInstance from SparkEnv and call the
> serialize
> > and deserialize methods. I was able to workaround it by making ClassTag
> > object in clojure, but it's less than ideal.
> >
> >
> > On Sun, Jun 1, 2014 at 4:25 PM, Matei Zaharia <matei.zaharia@gmail.com>
> > wrote:
> >
> >> BTW passing a ClassTag tells the Serializer what the type of object
> being
> >> serialized is when you compile your program, which will allow for more
> >> efficient serializers (especially on streams).
> >>
> >> Matei
> >>
> >> On Jun 1, 2014, at 4:24 PM, Matei Zaharia <matei.zaharia@gmail.com>
> wrote:
> >>
> >>> Why do you need to call Serializer from your own program? It’s an
> >> internal developer API so ideally it would only be called to extend
> Spark.
> >> Are you looking to implement a custom Serializer?
> >>>
> >>> Matei
> >>>
> >>> On Jun 1, 2014, at 3:40 PM, Soren Macbeth <soren@yieldbot.com> wrote:
> >>>
> >>>>
> >>
> https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/serializer/Serializer.scala#L64-L66
> >>>>
> >>>> These changes to the SerializerInstance make it really gross to call
> >>>> serialize and deserialize from non-scala languages. I'm not sure what
> >> the
> >>>> purpose of a ClassTag is, but if we could get some other arities that
> >> don't
> >>>> require classtags that would help a ton.
> >>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message