spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: ClassTag in Serializer in 1.0.0 makes non-scala callers sad panda
Date Sun, 01 Jun 2014 23:25:37 GMT
BTW passing a ClassTag tells the Serializer what the type of object being serialized is when
you compile your program, which will allow for more efficient serializers (especially on streams).

Matei

On Jun 1, 2014, at 4:24 PM, Matei Zaharia <matei.zaharia@gmail.com> wrote:

> Why do you need to call Serializer from your own program? It’s an internal developer
API so ideally it would only be called to extend Spark. Are you looking to implement a custom
Serializer?
> 
> Matei
> 
> On Jun 1, 2014, at 3:40 PM, Soren Macbeth <soren@yieldbot.com> wrote:
> 
>> https://github.com/apache/spark/blob/v1.0.0/core/src/main/scala/org/apache/spark/serializer/Serializer.scala#L64-L66
>> 
>> These changes to the SerializerInstance make it really gross to call
>> serialize and deserialize from non-scala languages. I'm not sure what the
>> purpose of a ClassTag is, but if we could get some other arities that don't
>> require classtags that would help a ton.
> 


Mime
View raw message